2023-09-29T22:58:26.819 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T22:58:26.820 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532 --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/orig.config.yaml 2023-09-29T22:58:26.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T22:58:26.928 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} email: yweinste@redhat.com first_in_suite: false job_id: '7406532' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCXEjO1dqSmbvU7ggLljVXfP71JORi5hJrFaY3B58mHbBN58u4n1UxhMi8w6e3hO+p5ubHQ/W04fNy2DD5Xew= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEM5Ub7/Hqh7BEtkSEpSolDqKk4UjAfv947YCtF6f6+zrXTNFwNmC2cxGfrRNbRbOoq17u1in5gC8LzFf0TVXvs= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T22:58:26.929 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-29T22:58:26.929 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-29T22:58:26.930 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T22:58:26.931 INFO:teuthology.task.internal:Checking packages... 2023-09-29T22:58:26.954 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-29T22:58:26.955 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T22:58:26.955 INFO:teuthology.packaging:ref: None 2023-09-29T22:58:26.955 INFO:teuthology.packaging:tag: None 2023-09-29T22:58:26.955 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:58:26.955 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T22:58:26.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:58:27.096 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-29T22:58:27.096 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T22:58:27.102 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T22:58:27.102 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T22:58:27.110 INFO:teuthology.task.internal:Saving configuration 2023-09-29T22:58:27.123 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T22:58:27.129 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T22:58:27.153 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:53:22.153335', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCXEjO1dqSmbvU7ggLljVXfP71JORi5hJrFaY3B58mHbBN58u4n1UxhMi8w6e3hO+p5ubHQ/W04fNy2DD5Xew='} 2023-09-29T22:58:27.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:53:22.151310', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEM5Ub7/Hqh7BEtkSEpSolDqKk4UjAfv947YCtF6f6+zrXTNFwNmC2cxGfrRNbRbOoq17u1in5gC8LzFf0TVXvs='} 2023-09-29T22:58:27.174 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T22:58:27.179 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2023-09-29T22:58:27.179 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-09-29T22:58:27.179 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T22:58:27.253 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f530506eee0>, signals=[15]) 2023-09-29T22:58:27.253 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T22:58:27.260 INFO:teuthology.task.internal:Opening connections... 2023-09-29T22:58:27.260 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T22:58:27.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:58:27.329 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T22:58:27.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:58:27.397 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T22:58:27.403 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2023-09-29T22:58:27.420 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2023-09-29T22:58:27.421 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2023-09-29T22:58:27.476 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Red Hat Enterprise Linux" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8.4 (Ootpa)" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:ID="rhel" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="fedora" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8.4" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.redhat.com/" 2023-09-29T22:58:27.477 INFO:teuthology.orchestra.run.smithi012.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-29T22:58:27.478 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-29T22:58:27.479 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2023-09-29T22:58:27.506 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2023-09-29T22:58:27.523 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2023-09-29T22:58:27.523 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2023-09-29T22:58:27.579 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Red Hat Enterprise Linux" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8.4 (Ootpa)" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:ID="rhel" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="fedora" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8.4" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.redhat.com/" 2023-09-29T22:58:27.580 INFO:teuthology.orchestra.run.smithi153.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-29T22:58:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-29T22:58:27.581 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-09-29T22:58:27.605 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T22:58:27.614 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T22:58:27.622 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T22:58:27.622 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2023-09-29T22:58:27.624 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2023-09-29T22:58:27.639 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T22:58:27.646 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T22:58:27.646 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2023-09-29T22:58:27.682 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2023-09-29T22:58:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-29T22:58:27.697 INFO:teuthology.orchestra.run.smithi153.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-29T22:58:27.699 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T22:58:27.753 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T22:58:27.769 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T22:58:27.769 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.f': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T22:58:27.769 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T22:58:27.770 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T22:58:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:58:27.785 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-09-29T22:58:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:58:27.786 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-09-29T22:58:27.841 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-29T22:58:27.842 INFO:teuthology.task.kernel:Running kernel on smithi153: 4.18.0-305.25.1.el8_4.x86_64 2023-09-29T22:58:27.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2023-09-29T22:58:27.843 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-29T22:58:27.843 INFO:teuthology.task.kernel:Running kernel on smithi012: 4.18.0-305.25.1.el8_4.x86_64 2023-09-29T22:58:27.843 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2023-09-29T22:58:29.265 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:29.265 INFO:teuthology.orchestra.run.smithi153.stdout:Unable to read consumer identity 2023-09-29T22:58:29.391 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:29.391 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-09-29T22:58:30.580 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-09-29T22:58:30.788 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 34 kB/s | 3.5 kB 00:00 2023-09-29T22:58:31.231 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-09-29T22:58:31.444 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-09-29T22:58:34.497 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2023-09-29T22:58:34.950 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2023-09-29T22:58:35.242 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-29T22:58:35.243 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-29T22:58:35.243 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-29T22:58:35.391 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T22:58:35.392 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T22:58:35.392 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T22:58:35.502 DEBUG:teuthology.orchestra.run.smithi153:> echo no | sudo yum reinstall kernel || true 2023-09-29T22:58:35.702 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-29T22:58:35.702 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-29T22:58:35.702 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-29T22:58:35.847 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:35.847 INFO:teuthology.orchestra.run.smithi153.stdout:Unable to read consumer identity 2023-09-29T22:58:35.855 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T22:58:35.856 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T22:58:35.856 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T22:58:35.955 DEBUG:teuthology.orchestra.run.smithi012:> echo no | sudo yum reinstall kernel || true 2023-09-29T22:58:36.137 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:02 ago on Fri 29 Sep 2023 10:58:34 PM UTC. 2023-09-29T22:58:36.279 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-29T22:58:36.279 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:36.279 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:36.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error: No packages marked for reinstall. 2023-09-29T22:58:36.320 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:36.320 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-09-29T22:58:36.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum reinstall -y kernel || true 2023-09-29T22:58:36.621 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:02 ago on Fri 29 Sep 2023 10:58:34 PM UTC. 2023-09-29T22:58:36.698 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:36.698 INFO:teuthology.orchestra.run.smithi153.stdout:Unable to read consumer identity 2023-09-29T22:58:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-29T22:58:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:36.850 INFO:teuthology.orchestra.run.smithi012.stderr:Error: No packages marked for reinstall. 2023-09-29T22:58:36.908 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum reinstall -y kernel || true 2023-09-29T22:58:36.980 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:02 ago on Fri 29 Sep 2023 10:58:34 PM UTC. 2023-09-29T22:58:37.111 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-29T22:58:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:37.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error: No packages marked for reinstall. 2023-09-29T22:58:37.182 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:58:37.272 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T22:58:37.272 INFO:teuthology.orchestra.run.smithi012.stdout:Unable to read consumer identity 2023-09-29T22:58:37.306 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.307 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.307 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.307 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T22:58:37.307 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T22:58:37.307 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2023-09-29T22:58:37.307 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T22:58:37.307 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:58:37.433 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2023-09-29T22:58:37.461 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2023-09-29T22:58:37.461 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2023-09-29T22:58:37.461 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.02 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 99.el8_4.1 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Group : System Environment/Base 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 9264631 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-29T22:58:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Relocations : (not relocatable) 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : Red Hat, Inc. 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : Red Hat, Inc. 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T22:58:37.463 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T22:58:37.464 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T22:58:37.464 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2023-09-29T22:58:37.464 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T22:58:37.464 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T22:58:37.465 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T22:58:37.465 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T22:58:37.573 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:03 ago on Fri 29 Sep 2023 10:58:34 PM UTC. 2023-09-29T22:58:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-29T22:58:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:37.748 INFO:teuthology.orchestra.run.smithi012.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-29T22:58:37.764 INFO:teuthology.orchestra.run.smithi012.stderr:Error: No packages marked for reinstall. 2023-09-29T22:58:37.814 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:58:37.944 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.944 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.945 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:37.945 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T22:58:37.945 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T22:58:37.945 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-29T22:58:37.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T22:58:37.945 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:58:38.077 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:38.077 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:38.078 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.02 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 99.el8_4.1 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2023-09-29T22:58:38.106 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Group : System Environment/Base 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 9264631 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Relocations : (not relocatable) 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : Red Hat, Inc. 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : Red Hat, Inc. 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T22:58:38.107 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T22:58:38.108 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T22:58:38.109 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T22:58:38.109 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:38.109 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T22:58:38.906 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2023-09-29T22:58:39.354 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2023-09-29T22:58:40.853 INFO:teuthology.orchestra.run.smithi153.stderr:done 2023-09-29T22:58:40.854 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-09-29T22:58:40.884 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.hZIPpUYliF 2023-09-29T22:58:40.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hZIPpUYliF 2023-09-29T22:58:40.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.hZIPpUYliF 2023-09-29T22:58:41.125 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.hZIPpUYliF is 6KB 2023-09-29T22:58:41.173 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.hZIPpUYliF 2023-09-29T22:58:41.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T22:58:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:done 2023-09-29T22:58:41.259 INFO:teuthology.orchestra.run.smithi153.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-29T22:58:41.259 INFO:teuthology.orchestra.run.smithi153.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-29T22:58:41.259 INFO:teuthology.orchestra.run.smithi153.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-29T22:58:41.259 INFO:teuthology.orchestra.run.smithi153.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-29T22:58:41.261 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T22:58:41.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:41.289 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.rGJTsGEbBW 2023-09-29T22:58:41.289 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rGJTsGEbBW 2023-09-29T22:58:41.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.rGJTsGEbBW 2023-09-29T22:58:41.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2023-09-29T22:58:41.526 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.rGJTsGEbBW is 6KB 2023-09-29T22:58:41.575 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.rGJTsGEbBW 2023-09-29T22:58:41.589 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T22:58:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-29T22:58:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-29T22:58:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-29T22:58:41.678 INFO:teuthology.orchestra.run.smithi012.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-29T22:58:41.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-29T22:58:41.798 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2023-09-29T22:59:11.454 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2023-09-29T22:59:11.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T22:59:11.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:59:11.801 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-29T22:59:11.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-09-29T22:59:11.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:59:45.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-09-29T22:59:54.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T22:59:54.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:00:11.819 DEBUG:teuthology.orchestra.remote:timed out 2023-09-29T23:00:13.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-09-29T23:00:20.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-09-29T23:00:20.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:00:21.145 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-09-29T23:00:21.460 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-09-29T23:00:21.460 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-29T23:00:21.461 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-09-29T23:00:21.475 INFO:teuthology.orchestra.run.smithi012.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-29T23:00:21.475 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-29T23:00:21.475 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T23:00:21.475 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-29T23:00:22.476 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-29T23:00:22.477 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T23:00:22.601 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2023-09-29T23:00:22.612 DEBUG:teuthology.parallel:result is None 2023-09-29T23:00:25.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T23:00:25.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:00:26.060 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-09-29T23:00:26.402 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T23:00:26.402 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-29T23:00:26.402 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-09-29T23:00:26.416 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-29T23:00:26.416 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-29T23:00:26.416 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T23:00:26.417 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-29T23:00:27.417 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2023-09-29T23:00:27.418 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T23:00:27.532 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2023-09-29T23:00:27.544 DEBUG:teuthology.parallel:result is None 2023-09-29T23:00:27.545 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T23:00:27.552 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T23:00:27.552 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T23:00:27.555 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T23:00:27.570 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T23:00:27.577 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T23:00:27.584 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T23:00:27.584 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T23:00:27.612 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T23:00:27.639 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T23:00:27.645 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T23:00:27.645 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T23:00:27.676 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T23:00:27.706 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:00:27.717 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:00:27.734 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:00:27.745 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:00:27.747 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T23:00:27.753 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T23:00:27.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T23:00:27.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T23:00:27.787 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T23:00:27.795 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T23:00:27.796 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T23:00:27.827 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T23:00:27.843 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:00:27.923 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:00:27.962 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:00:28.045 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:00:28.084 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:00:28.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T23:00:28.150 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:00:28.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:00:28.245 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:00:28.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:00:28.368 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:00:28.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T23:00:28.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2023-09-29T23:00:28.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2023-09-29T23:00:28.497 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:00:28.563 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:00:28.825 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T23:00:28.834 INFO:teuthology.task.internal:Starting timer... 2023-09-29T23:00:28.834 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T23:00:28.852 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T23:00:28.860 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-29T23:00:28.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2023-09-29T23:00:28.924 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: [ OK ] 2023-09-29T23:00:28.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2023-09-29T23:00:28.972 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: [ OK ] 2023-09-29T23:00:28.973 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T23:00:28.974 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2023-09-29T23:00:28.999 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2023-09-29T23:00:28.999 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2023-09-29T23:00:29.024 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2023-09-29T23:00:29.024 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2023-09-29T23:00:29.025 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:00:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:00:29.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:00:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:00:29.091 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T23:00:29.091 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2023-09-29T23:00:29.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2023-09-29T23:00:29.162 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T23:00:29.171 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-29T23:00:29.171 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T23:00:29.188 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T23:00:29.190 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T23:08:48.196 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2023-09-29T23:08:48.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-09-29T23:08:48.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:08:48.267 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-09-29T23:08:48.291 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-09-29T23:08:48.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T23:08:48.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:08:48.361 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-09-29T23:08:48.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-09-29T23:08:48.430 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T23:08:48.440 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T23:08:48.440 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:08:48.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:08:48.443 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:08:48.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:08:48.472 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:08:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:08:48.498 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:08:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:08:48.525 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2023-09-29T23:08:48.536 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2023-09-29T23:08:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2023-09-29T23:08:48.551 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:08:48.554 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2023-09-29T23:08:48.565 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:08:48.567 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:08:48.581 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:08:48.637 INFO:teuthology.orchestra.run.smithi012.stderr:bash: ntpq: command not found 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:210 Number of sources = 4 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.640 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.646 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:210 Number of sources = 4 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:08:48.650 INFO:teuthology.run_tasks:Running task install... 2023-09-29T23:08:48.659 DEBUG:teuthology.task.install:project ceph 2023-09-29T23:08:48.659 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-29T23:08:48.659 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-29T23:08:48.659 INFO:teuthology.task.install:Using flavor: default 2023-09-29T23:08:48.669 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-29T23:08:48.669 INFO:teuthology.task.install:extra packages: [] 2023-09-29T23:08:48.670 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:08:48.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:08:48.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:08:48.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:08:48.861 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:08:48.861 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:08:48.999 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:08:49.000 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:08:49.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:08:49.027 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T23:08:49.027 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:08:49.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:08:49.171 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:08:49.237 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2023-09-29T23:08:49.239 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2023-09-29T23:08:49.651 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:08:49.953 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:08:49.953 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:08:49.973 INFO:teuthology.orchestra.run.smithi012.stdout:44 files removed 2023-09-29T23:08:50.031 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw 2023-09-29T23:08:50.158 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:08:50.158 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:08:50.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:08:50.189 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T23:08:50.189 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:08:50.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:08:50.336 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:08:50.400 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:08:50.402 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2023-09-29T23:08:50.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2023-09-29T23:08:50.799 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:08:51.097 INFO:teuthology.orchestra.run.smithi153.stdout:44 files removed 2023-09-29T23:08:51.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2023-09-29T23:08:51.251 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 197 kB/s | 78 kB 00:00 2023-09-29T23:08:51.499 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:08:51.547 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 36 kB/s | 9.2 kB 00:00 2023-09-29T23:08:52.825 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 94 kB/s | 78 kB 00:00 2023-09-29T23:08:52.999 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 1.0 kB/s | 1.5 kB 00:01 2023-09-29T23:08:53.113 INFO:teuthology.orchestra.run.smithi012.stdout:Copr repo for python3-asyncssh owned by ceph 43 kB/s | 3.5 kB 00:00 2023-09-29T23:08:53.482 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 15 kB/s | 9.2 kB 00:00 2023-09-29T23:08:54.231 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-09-29T23:08:55.047 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 975 B/s | 1.5 kB 00:01 2023-09-29T23:08:55.155 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 45 kB/s | 3.5 kB 00:00 2023-09-29T23:08:55.882 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-09-29T23:08:57.744 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-09-29T23:08:58.233 INFO:teuthology.orchestra.run.smithi012.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 95 MB/s | 38 MB 00:00 2023-09-29T23:08:59.153 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-09-29T23:08:59.533 INFO:teuthology.orchestra.run.smithi153.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 110 MB/s | 33 MB 00:00 2023-09-29T23:09:04.273 INFO:teuthology.orchestra.run.smithi012.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 MB/s | 33 MB 00:00 2023-09-29T23:09:04.717 INFO:teuthology.orchestra.run.smithi153.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 105 MB/s | 38 MB 00:00 2023-09-29T23:09:13.134 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:09:13.136 INFO:teuthology.orchestra.run.smithi012.stdout:==================================================================================================== 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout:==================================================================================================== 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:09:13.137 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-29T23:09:13.138 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:09:13.139 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout:==================================================================================================== 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout:Install 21 Packages 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 2 Packages 2023-09-29T23:09:13.140 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:09:13.141 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 63 M 2023-09-29T23:09:13.141 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:09:13.536 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout:==================================================================================================== 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout:==================================================================================================== 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:09:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-29T23:09:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:09:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:==================================================================================================== 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:Install 21 Packages 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:09:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 63 M 2023-09-29T23:09:13.543 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:09:17.963 INFO:teuthology.orchestra.run.smithi012.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.1 MB/s | 5.2 MB 00:04 2023-09-29T23:09:18.021 INFO:teuthology.orchestra.run.smithi153.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.2 MB/s | 5.2 MB 00:04 2023-09-29T23:09:18.034 INFO:teuthology.orchestra.run.smithi012.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 346 kB/s | 24 kB 00:00 2023-09-29T23:09:18.076 INFO:teuthology.orchestra.run.smithi153.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 446 kB/s | 24 kB 00:00 2023-09-29T23:09:18.441 INFO:teuthology.orchestra.run.smithi153.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.1 MB/s | 796 kB 00:00 2023-09-29T23:09:18.474 INFO:teuthology.orchestra.run.smithi012.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 1.8 MB/s | 796 kB 00:00 2023-09-29T23:09:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.7 MB/s | 506 kB 00:00 2023-09-29T23:09:18.791 INFO:teuthology.orchestra.run.smithi012.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.6 MB/s | 506 kB 00:00 2023-09-29T23:09:19.353 INFO:teuthology.orchestra.run.smithi012.stdout:(5/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 3.9 MB/s | 23 MB 00:06 2023-09-29T23:09:19.444 INFO:teuthology.orchestra.run.smithi012.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 494 kB/s | 45 kB 00:00 2023-09-29T23:09:19.538 INFO:teuthology.orchestra.run.smithi012.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.2 MB/s | 115 kB 00:00 2023-09-29T23:09:19.625 INFO:teuthology.orchestra.run.smithi012.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.2 MB/s | 190 kB 00:00 2023-09-29T23:09:19.799 INFO:teuthology.orchestra.run.smithi012.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 2.1 MB/s | 370 kB 00:00 2023-09-29T23:09:19.952 INFO:teuthology.orchestra.run.smithi012.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.3 MB/s | 358 kB 00:00 2023-09-29T23:09:20.029 INFO:teuthology.orchestra.run.smithi012.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.3 MB/s | 107 kB 00:00 2023-09-29T23:09:20.202 INFO:teuthology.orchestra.run.smithi012.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2023-09-29T23:09:20.220 INFO:teuthology.orchestra.run.smithi012.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 3.3 MB/s | 59 kB 00:00 2023-09-29T23:09:20.238 INFO:teuthology.orchestra.run.smithi012.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 4.1 MB/s | 75 kB 00:00 2023-09-29T23:09:20.683 INFO:teuthology.orchestra.run.smithi012.stdout:(15/23): thrift-0.13.0-2.el8.x86_64.rpm 3.9 MB/s | 1.7 MB 00:00 2023-09-29T23:09:20.722 INFO:teuthology.orchestra.run.smithi012.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2023-09-29T23:09:20.739 INFO:teuthology.orchestra.run.smithi012.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 2.6 MB/s | 44 kB 00:00 2023-09-29T23:09:20.758 INFO:teuthology.orchestra.run.smithi012.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 13 MB/s | 259 kB 00:00 2023-09-29T23:09:20.778 INFO:teuthology.orchestra.run.smithi012.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 18 MB/s | 353 kB 00:00 2023-09-29T23:09:20.867 INFO:teuthology.orchestra.run.smithi153.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 2.6 MB/s | 5.6 MB 00:02 2023-09-29T23:09:20.897 INFO:teuthology.orchestra.run.smithi153.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 1.5 MB/s | 45 kB 00:00 2023-09-29T23:09:20.928 INFO:teuthology.orchestra.run.smithi153.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 3.6 MB/s | 115 kB 00:00 2023-09-29T23:09:20.989 INFO:teuthology.orchestra.run.smithi012.stdout:(20/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 2.5 MB/s | 5.6 MB 00:02 2023-09-29T23:09:20.995 INFO:teuthology.orchestra.run.smithi153.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.8 MB/s | 190 kB 00:00 2023-09-29T23:09:21.107 INFO:teuthology.orchestra.run.smithi153.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 3.2 MB/s | 370 kB 00:00 2023-09-29T23:09:21.217 INFO:teuthology.orchestra.run.smithi153.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 3.2 MB/s | 358 kB 00:00 2023-09-29T23:09:21.253 INFO:teuthology.orchestra.run.smithi153.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.9 MB/s | 107 kB 00:00 2023-09-29T23:09:21.400 INFO:teuthology.orchestra.run.smithi012.stdout:(21/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 5.6 MB/s | 3.5 MB 00:00 2023-09-29T23:09:21.623 INFO:teuthology.orchestra.run.smithi153.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 829 kB/s | 306 kB 00:00 2023-09-29T23:09:21.684 INFO:teuthology.orchestra.run.smithi153.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 971 kB/s | 59 kB 00:00 2023-09-29T23:09:21.740 INFO:teuthology.orchestra.run.smithi153.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-09-29T23:09:21.829 INFO:teuthology.orchestra.run.smithi012.stdout:(22/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 2.0 MB/s | 17 MB 00:08 2023-09-29T23:09:21.840 INFO:teuthology.orchestra.run.smithi012.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 4.2 MB/s | 3.6 MB 00:00 2023-09-29T23:09:21.844 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:09:21.845 INFO:teuthology.orchestra.run.smithi012.stdout:Total 7.3 MB/s | 63 MB 00:08 2023-09-29T23:09:21.850 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:09:21.899 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:09:21.900 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:09:21.927 INFO:teuthology.orchestra.run.smithi153.stdout:(15/23): thrift-0.13.0-2.el8.x86_64.rpm 9.3 MB/s | 1.7 MB 00:00 2023-09-29T23:09:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.2 MB/s | 44 kB 00:00 2023-09-29T23:09:21.982 INFO:teuthology.orchestra.run.smithi153.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-09-29T23:09:22.001 INFO:teuthology.orchestra.run.smithi153.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-09-29T23:09:22.014 INFO:teuthology.orchestra.run.smithi153.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.5 MB/s | 47 kB 00:00 2023-09-29T23:09:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:(20/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 1.9 MB/s | 17 MB 00:08 2023-09-29T23:09:22.609 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:09:22.629 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:09:22.642 INFO:teuthology.orchestra.run.smithi153.stdout:(21/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 5.5 MB/s | 3.5 MB 00:00 2023-09-29T23:09:22.817 INFO:teuthology.orchestra.run.smithi153.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 6.5 MB/s | 3.6 MB 00:00 2023-09-29T23:09:23.282 INFO:teuthology.orchestra.run.smithi153.stdout:(23/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 2.4 MB/s | 23 MB 00:09 2023-09-29T23:09:23.286 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:09:23.286 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.5 MB/s | 63 MB 00:09 2023-09-29T23:09:23.290 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:09:23.335 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:09:23.335 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:09:23.740 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:09:23.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-29T23:09:24.078 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:09:24.083 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:09:24.339 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-29T23:09:24.549 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:24.642 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:24.831 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:09:25.079 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-29T23:09:25.288 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:09:25.299 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:25.507 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-29T23:09:25.721 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:25.882 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-29T23:09:25.908 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:09:26.006 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:09:26.115 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:26.213 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:26.216 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-29T23:09:26.269 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-29T23:09:26.421 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:09:26.598 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-09-29T23:09:26.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-29T23:09:26.890 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:27.293 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-29T23:09:27.316 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:27.457 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:27.524 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:27.532 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:09:27.599 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:09:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-29T23:09:27.818 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-29T23:09:27.827 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:09:28.049 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-29T23:09:28.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:09:28.254 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-29T23:09:28.297 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-29T23:09:28.441 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:09:28.513 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:09:28.684 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-29T23:09:28.886 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-29T23:09:29.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-29T23:09:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-29T23:09:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-29T23:09:29.331 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:29.405 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-29T23:09:29.743 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:09:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:09:30.269 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-29T23:09:30.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:09:30.527 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:09:30.733 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-29T23:09:30.950 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-29T23:09:31.203 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-29T23:09:31.415 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-29T23:09:34.697 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:34.914 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:35.539 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:35.802 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:09:35.893 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:09:36.039 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:37.139 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:37.406 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:38.022 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:09:38.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:09:38.430 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:09:38.547 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:54.464 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:54.464 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /dev 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /run 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /dev 2023-09-29T23:09:54.465 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:09:56.248 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:09:56.332 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:09:56.332 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /run 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2023-09-29T23:09:56.490 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2023-09-29T23:09:56.491 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2023-09-29T23:09:56.491 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2023-09-29T23:09:56.491 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2023-09-29T23:09:56.491 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:09:56.536 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:09:56.536 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-29T23:09:57.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-29T23:09:57.319 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-29T23:09:57.320 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-29T23:09:58.198 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:09:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:09:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:09:58.496 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:09:58.496 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:09:58.646 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:09:58.646 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:09:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-29T23:09:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-29T23:09:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-29T23:10:00.644 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:10:00.765 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:00.765 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2023-09-29T23:10:00.765 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.765 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.766 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:00.767 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:00.869 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:10:00.869 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:00.898 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-test 2023-09-29T23:10:01.285 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:02.075 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:10:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.898 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-29T23:10:02.900 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:02.900 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:10:03.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2023-09-29T23:10:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:04.034 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:10:04.035 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-09-29T23:10:04.035 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:10:04.035 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-09-29T23:10:04.035 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:10:04.035 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-09-29T23:10:04.036 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2023-09-29T23:10:04.037 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:04.037 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 53 M 2023-09-29T23:10:04.037 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 233 M 2023-09-29T23:10:04.037 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:10:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-09-29T23:10:04.081 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2023-09-29T23:10:04.099 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 10 MB/s | 187 kB 00:00 2023-09-29T23:10:04.119 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-09-29T23:10:04.298 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 672 kB/s | 175 kB 00:00 2023-09-29T23:10:04.361 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:06.194 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:10:06.195 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-09-29T23:10:06.195 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:10:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2023-09-29T23:10:06.197 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2023-09-29T23:10:06.197 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:06.197 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 53 M 2023-09-29T23:10:06.197 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 233 M 2023-09-29T23:10:06.197 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:10:06.201 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-09-29T23:10:06.251 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.2 MB/s | 161 kB 00:00 2023-09-29T23:10:06.274 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.8 MB/s | 187 kB 00:00 2023-09-29T23:10:06.294 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-09-29T23:10:06.416 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 22 MB/s | 52 MB 00:02 2023-09-29T23:10:06.420 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:06.420 INFO:teuthology.orchestra.run.smithi012.stdout:Total 22 MB/s | 53 MB 00:02 2023-09-29T23:10:06.425 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:10:06.450 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:10:06.451 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:10:06.459 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 667 kB/s | 175 kB 00:00 2023-09-29T23:10:06.815 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:10:06.820 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:10:07.329 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:10:07.550 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-29T23:10:07.773 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-29T23:10:07.854 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-29T23:10:08.012 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-29T23:10:08.173 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-29T23:10:08.314 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:10:08.929 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 52 MB 00:02 2023-09-29T23:10:08.932 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:08.932 INFO:teuthology.orchestra.run.smithi153.stdout:Total 19 MB/s | 53 MB 00:02 2023-09-29T23:10:08.937 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:10:08.960 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:10:08.960 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:10:09.349 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:10:09.354 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:10:09.890 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:10:10.120 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-29T23:10:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-29T23:10:10.421 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-29T23:10:10.642 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-29T23:10:10.866 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-29T23:10:11.041 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:10:13.599 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:10:13.854 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:10:14.147 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:10:14.147 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-29T23:10:14.147 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-29T23:10:14.147 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-29T23:10:14.147 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-29T23:10:14.148 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-29T23:10:15.270 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-29T23:10:15.270 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:16.061 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:10:16.381 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-29T23:10:16.658 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-29T23:10:17.244 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.5-12.el8.x86_64 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:17.350 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-29T23:10:17.351 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-29T23:10:17.351 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-29T23:10:17.351 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:17.351 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:17.465 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph 2023-09-29T23:10:17.744 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-29T23:10:17.744 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:17.839 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:18.668 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:19.773 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.5-12.el8.x86_64 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-29T23:10:19.872 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-29T23:10:19.873 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:19.873 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:10:19.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2023-09-29T23:10:20.345 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:20.593 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:10:20.596 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================= 2023-09-29T23:10:20.596 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:10:20.596 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================= 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-29T23:10:20.597 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-29T23:10:20.598 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-29T23:10:20.599 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================= 2023-09-29T23:10:20.600 INFO:teuthology.orchestra.run.smithi012.stdout:Install 28 Packages 2023-09-29T23:10:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:20.601 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 27 M 2023-09-29T23:10:20.601 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 106 M 2023-09-29T23:10:20.601 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:10:20.813 INFO:teuthology.orchestra.run.smithi012.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-09-29T23:10:20.897 INFO:teuthology.orchestra.run.smithi012.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 6.7 MB/s | 1.6 MB 00:00 2023-09-29T23:10:21.220 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:21.319 INFO:teuthology.orchestra.run.smithi012.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.5 MB/s | 2.3 MB 00:00 2023-09-29T23:10:21.396 INFO:teuthology.orchestra.run.smithi012.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 3.0 MB/s | 238 kB 00:00 2023-09-29T23:10:21.474 INFO:teuthology.orchestra.run.smithi012.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 566 kB/s | 44 kB 00:00 2023-09-29T23:10:21.535 INFO:teuthology.orchestra.run.smithi012.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.0 MB/s | 185 kB 00:00 2023-09-29T23:10:21.565 INFO:teuthology.orchestra.run.smithi012.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 5.6 MB/s | 173 kB 00:00 2023-09-29T23:10:21.607 INFO:teuthology.orchestra.run.smithi012.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 8.9 MB/s | 384 kB 00:00 2023-09-29T23:10:21.626 INFO:teuthology.orchestra.run.smithi012.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 583 kB/s | 11 kB 00:00 2023-09-29T23:10:21.646 INFO:teuthology.orchestra.run.smithi012.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 918 kB/s | 18 kB 00:00 2023-09-29T23:10:21.760 INFO:teuthology.orchestra.run.smithi012.stdout:(11/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 19 MB/s | 17 MB 00:00 2023-09-29T23:10:21.761 INFO:teuthology.orchestra.run.smithi012.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 424 kB/s | 49 kB 00:00 2023-09-29T23:10:21.786 INFO:teuthology.orchestra.run.smithi012.stdout:(13/28): python3-pecan-1.3.2-9.el8.noarch.rpm 11 MB/s | 283 kB 00:00 2023-09-29T23:10:21.805 INFO:teuthology.orchestra.run.smithi012.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 810 kB/s | 16 kB 00:00 2023-09-29T23:10:21.825 INFO:teuthology.orchestra.run.smithi012.stdout:(15/28): python3-simplegeneric-0.8.1-17.el8.noa 977 kB/s | 19 kB 00:00 2023-09-29T23:10:21.844 INFO:teuthology.orchestra.run.smithi012.stdout:(16/28): python3-more-itertools-7.2.0-3.el8.noa 699 kB/s | 59 kB 00:00 2023-09-29T23:10:21.845 INFO:teuthology.orchestra.run.smithi012.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2023-09-29T23:10:21.867 INFO:teuthology.orchestra.run.smithi012.stdout:(18/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 3.7 MB/s | 3.9 MB 00:01 2023-09-29T23:10:21.868 INFO:teuthology.orchestra.run.smithi012.stdout:(19/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.2 MB/s | 29 kB 00:00 2023-09-29T23:10:21.871 INFO:teuthology.orchestra.run.smithi012.stdout:(20/28): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2023-09-29T23:10:21.892 INFO:teuthology.orchestra.run.smithi012.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 10 MB/s | 251 kB 00:00 2023-09-29T23:10:21.911 INFO:teuthology.orchestra.run.smithi012.stdout:(22/28): python3-zc-lockfile-2.0-2.el8.noarch.r 1.2 MB/s | 23 kB 00:00 2023-09-29T23:10:21.914 INFO:teuthology.orchestra.run.smithi012.stdout:(23/28): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2023-09-29T23:10:21.968 INFO:teuthology.orchestra.run.smithi012.stdout:(24/28): python3-markupsafe-0.23-19.el8.x86_64. 714 kB/s | 39 kB 00:00 2023-09-29T23:10:21.991 INFO:teuthology.orchestra.run.smithi012.stdout:(25/28): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-09-29T23:10:22.002 INFO:teuthology.orchestra.run.smithi012.stdout:(26/28): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2023-09-29T23:10:22.005 INFO:teuthology.orchestra.run.smithi012.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 4.2 MB/s | 157 kB 00:00 2023-09-29T23:10:22.045 INFO:teuthology.orchestra.run.smithi012.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2023-09-29T23:10:22.046 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:22.047 INFO:teuthology.orchestra.run.smithi012.stdout:Total 19 MB/s | 27 MB 00:01 2023-09-29T23:10:22.051 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:10:22.090 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:10:22.090 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:10:22.468 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:10:22.472 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:10:23.045 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================= 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================= 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-29T23:10:23.048 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-29T23:10:23.049 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-29T23:10:23.050 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================= 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout:Install 28 Packages 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:23.051 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2023-09-29T23:10:23.052 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 106 M 2023-09-29T23:10:23.052 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:10:23.239 INFO:teuthology.orchestra.run.smithi153.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2023-09-29T23:10:23.344 INFO:teuthology.orchestra.run.smithi153.stdout:(2/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 8.9 MB/s | 2.3 MB 00:00 2023-09-29T23:10:23.492 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:10:23.739 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-29T23:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout:(3/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.0 MB/s | 1.6 MB 00:00 2023-09-29T23:10:23.944 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-29T23:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.4 MB/s | 238 kB 00:00 2023-09-29T23:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout:(5/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 21 MB/s | 17 MB 00:00 2023-09-29T23:10:24.166 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-29T23:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 206 kB/s | 44 kB 00:00 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.2 MB/s | 173 kB 00:00 2023-09-29T23:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout:(8/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.1 MB/s | 185 kB 00:00 2023-09-29T23:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout:(9/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 4.3 MB/s | 384 kB 00:00 2023-09-29T23:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 617 kB/s | 11 kB 00:00 2023-09-29T23:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 995 kB/s | 18 kB 00:00 2023-09-29T23:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 2.6 MB/s | 49 kB 00:00 2023-09-29T23:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 3.1 MB/s | 59 kB 00:00 2023-09-29T23:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 903 kB/s | 16 kB 00:00 2023-09-29T23:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout:(15/28): python3-simplegeneric-0.8.1-17.el8.noa 1.0 MB/s | 19 kB 00:00 2023-09-29T23:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout:(16/28): python3-pecan-1.3.2-9.el8.noarch.rpm 3.9 MB/s | 283 kB 00:00 2023-09-29T23:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 1.3 MB/s | 24 kB 00:00 2023-09-29T23:10:24.408 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-29T23:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.6 MB/s | 29 kB 00:00 2023-09-29T23:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 1.5 MB/s | 27 kB 00:00 2023-09-29T23:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout:(20/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.6 MB/s | 251 kB 00:00 2023-09-29T23:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout:(21/28): python3-waitress-1.4.3-1.el8.noarch.rp 3.5 MB/s | 255 kB 00:00 2023-09-29T23:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 2.4 MB/s | 85 kB 00:00 2023-09-29T23:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 1.2 MB/s | 23 kB 00:00 2023-09-29T23:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout:(24/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 2.9 MB/s | 3.9 MB 00:01 2023-09-29T23:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout:(25/28): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-09-29T23:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout:(26/28): python3-markupsafe-0.23-19.el8.x86_64. 271 kB/s | 39 kB 00:00 2023-09-29T23:10:24.728 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-29T23:10:24.754 INFO:teuthology.orchestra.run.smithi153.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 868 kB/s | 157 kB 00:00 2023-09-29T23:10:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 344 kB/s | 54 kB 00:00 2023-09-29T23:10:24.793 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:24.793 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 27 MB 00:01 2023-09-29T23:10:24.797 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:10:24.833 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:10:24.833 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:10:24.937 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-29T23:10:25.162 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-29T23:10:25.216 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:10:25.221 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:10:25.370 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-29T23:10:25.552 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-29T23:10:25.708 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-29T23:10:25.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-29T23:10:26.064 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-29T23:10:26.209 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-29T23:10:26.395 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:10:26.413 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-29T23:10:26.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-29T23:10:26.670 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-29T23:10:26.792 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-29T23:10:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-29T23:10:27.033 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-29T23:10:27.127 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-29T23:10:27.261 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-29T23:10:27.423 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-29T23:10:27.531 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-29T23:10:27.750 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-29T23:10:27.830 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-29T23:10:28.006 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-29T23:10:28.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-29T23:10:28.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-29T23:10:28.406 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-29T23:10:28.489 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-29T23:10:28.679 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-29T23:10:28.755 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-29T23:10:28.826 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-29T23:10:28.899 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-29T23:10:29.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-29T23:10:29.283 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-29T23:10:29.469 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-29T23:10:29.712 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-29T23:10:29.959 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-29T23:10:30.119 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-29T23:10:30.402 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-29T23:10:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-29T23:10:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-29T23:10:30.652 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-29T23:10:30.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-29T23:10:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-29T23:10:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-29T23:10:31.228 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-29T23:10:31.402 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-29T23:10:31.552 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-29T23:10:31.571 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-29T23:10:31.685 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-29T23:10:32.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-29T23:10:32.105 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-29T23:10:32.106 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-29T23:10:32.107 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-29T23:10:32.108 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-29T23:10:32.108 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-29T23:10:32.108 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-29T23:10:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-29T23:10:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-29T23:10:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-29T23:10:33.659 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-29T23:10:33.659 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:34.427 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-29T23:10:34.576 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-29T23:10:34.973 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-29T23:10:35.113 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-29T23:10:35.332 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-29T23:10:35.513 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-29T23:10:35.531 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-29T23:10:35.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-29T23:10:35.878 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:10:35.988 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:35.988 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-29T23:10:35.989 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-2.6-1.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-29T23:10:35.990 INFO:teuthology.orchestra.run.smithi012.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:35.991 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-29T23:10:36.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-29T23:10:36.066 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-29T23:10:36.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-29T23:10:36.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-base 2023-09-29T23:10:36.530 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:37.367 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:37.766 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-29T23:10:37.766 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:38.767 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:10:39.263 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:10:39.264 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:10:39.264 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:39.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install cephadm 2023-09-29T23:10:39.739 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:39.781 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:10:39.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:39.882 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:10:39.882 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-29T23:10:39.883 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-2.6-1.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-29T23:10:39.884 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:39.885 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:10:40.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2023-09-29T23:10:40.425 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:40.649 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:41.308 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:42.588 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 80 k 2023-09-29T23:10:42.589 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 363 k 2023-09-29T23:10:42.590 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:10:42.678 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:10:42.811 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 361 kB/s | 80 kB 00:00 2023-09-29T23:10:42.812 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:42.812 INFO:teuthology.orchestra.run.smithi012.stdout:Total 359 kB/s | 80 kB 00:00 2023-09-29T23:10:42.812 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:10:42.821 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:10:42.821 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:10:42.823 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:10:42.828 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:10:42.872 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:10:43.171 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:10:43.172 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:10:43.172 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:10:43.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2023-09-29T23:10:43.635 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:44.491 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:46.353 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:10:46.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:46.355 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 80 k 2023-09-29T23:10:46.355 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 363 k 2023-09-29T23:10:46.355 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:10:46.484 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 616 kB/s | 80 kB 00:00 2023-09-29T23:10:46.485 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:46.485 INFO:teuthology.orchestra.run.smithi153.stdout:Total 608 kB/s | 80 kB 00:00 2023-09-29T23:10:46.486 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:10:46.494 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:10:46.494 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:10:46.496 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:10:46.500 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:10:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:10:46.821 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:46.975 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:47.185 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:48.120 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:48.120 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:50.002 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:10:50.107 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:50.107 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:10:50.108 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:10:50.108 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:50.108 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:50.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T23:10:50.627 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:50.847 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:51.028 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:51.232 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:51.428 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:52.238 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:10:52.238 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:53.402 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:10:53.403 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:53.403 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-09-29T23:10:53.403 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:53.403 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:10:53.403 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 161 k 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 490 k 2023-09-29T23:10:53.404 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:10:53.558 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 1.0 MB/s | 161 kB 00:00 2023-09-29T23:10:53.558 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:53.558 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.0 MB/s | 161 kB 00:00 2023-09-29T23:10:53.559 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:10:53.573 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:10:53.573 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:10:53.629 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:10:53.633 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:10:53.752 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:10:53.896 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:54.178 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:10:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:54.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:10:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:10:54.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:54.274 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:10:54.325 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:54.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T23:10:54.777 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:55.326 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:55.326 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:55.709 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:57.492 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:10:57.549 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:10:57.550 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:10:57.551 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:10:57.551 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:10:57.551 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 161 k 2023-09-29T23:10:57.551 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 490 k 2023-09-29T23:10:57.551 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:10:57.598 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:57.598 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:10:57.598 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:10:57.598 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:10:57.599 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:10:57.719 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr 2023-09-29T23:10:57.808 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 627 kB/s | 161 kB 00:00 2023-09-29T23:10:57.808 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:10:57.809 INFO:teuthology.orchestra.run.smithi153.stdout:Total 624 kB/s | 161 kB 00:00 2023-09-29T23:10:57.809 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:10:57.822 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:10:57.822 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:10:57.876 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:10:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:10:57.995 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:10:58.106 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:10:58.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:58.578 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:58.925 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:10:59.576 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:10:59.577 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:00.354 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:11:00.855 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:00.856 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:11:00.856 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:00.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T23:11:01.334 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:01.481 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:11:01.578 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:01.578 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:11:01.578 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:11:01.578 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:01.578 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:01.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2023-09-29T23:11:02.068 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:02.221 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:02.893 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:04.143 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:04.144 INFO:teuthology.orchestra.run.smithi012.stdout:=================================================================================================== 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout:=================================================================================================== 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:04.145 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout:=================================================================================================== 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 2.0 M 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 8.0 M 2023-09-29T23:11:04.146 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:11:04.204 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:11:04.320 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 210 kB/s | 23 kB 00:00 2023-09-29T23:11:04.349 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 1.2 MB/s | 34 kB 00:00 2023-09-29T23:11:04.380 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): python3-routes-2.4.1-12.el8.noarch.rpm 6.1 MB/s | 196 kB 00:00 2023-09-29T23:11:04.398 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 77 kB/s | 14 kB 00:00 2023-09-29T23:11:04.412 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-09-29T23:11:04.455 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 6.9 MB/s | 1.7 MB 00:00 2023-09-29T23:11:04.457 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:04.457 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.4 MB/s | 2.0 MB 00:00 2023-09-29T23:11:04.459 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:11:04.468 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:11:04.468 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:11:04.561 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:11:04.566 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:11:04.681 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:04.682 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:11:04.682 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:04.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T23:11:04.824 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:11:05.023 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-29T23:11:05.151 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:05.213 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-29T23:11:05.430 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-29T23:11:05.593 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-29T23:11:05.986 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-29T23:11:06.035 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:06.190 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:11:06.486 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:11:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-29T23:11:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-29T23:11:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-29T23:11:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-29T23:11:06.487 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-29T23:11:07.461 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-29T23:11:07.461 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:07.916 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout:=================================================================================================== 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout:=================================================================================================== 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-29T23:11:07.917 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:07.918 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:11:07.918 INFO:teuthology.orchestra.run.smithi153.stdout:=================================================================================================== 2023-09-29T23:11:07.918 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2023-09-29T23:11:07.918 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:07.918 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2023-09-29T23:11:07.919 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 8.0 M 2023-09-29T23:11:07.919 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:11:08.183 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 209 kB/s | 23 kB 00:00 2023-09-29T23:11:08.262 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 76 kB/s | 14 kB 00:00 2023-09-29T23:11:08.309 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 267 kB/s | 34 kB 00:00 2023-09-29T23:11:08.340 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2023-09-29T23:11:08.456 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-09-29T23:11:08.485 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 4.1 MB/s | 1.7 MB 00:00 2023-09-29T23:11:08.487 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:08.487 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2023-09-29T23:11:08.489 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:11:08.497 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:11:08.497 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:11:08.584 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:11:08.589 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:11:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:11:09.085 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-29T23:11:09.316 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-29T23:11:09.505 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:11:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:09.610 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-29T23:11:09.611 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-29T23:11:09.611 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-29T23:11:09.611 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:09.611 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-29T23:11:09.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T23:11:10.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-29T23:11:10.113 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:11:10.571 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:11:10.571 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-29T23:11:10.572 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-29T23:11:10.572 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-29T23:11:10.572 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-29T23:11:10.572 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-29T23:11:11.069 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:11.653 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-29T23:11:11.653 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:13.029 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:13.030 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================================== 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================================== 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-29T23:11:13.031 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================================== 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout:Install 8 Packages 2023-09-29T23:11:13.032 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:13.033 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 36 M 2023-09-29T23:11:13.033 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 199 M 2023-09-29T23:11:13.033 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:11:13.085 INFO:teuthology.orchestra.run.smithi012.stdout:(1/8): libgfortran-8.4.1-1.el8.x86_64.rpm 12 MB/s | 639 kB 00:00 2023-09-29T23:11:13.090 INFO:teuthology.orchestra.run.smithi012.stdout:(2/8): libquadmath-8.4.1-1.el8.x86_64.rpm 2.9 MB/s | 169 kB 00:00 2023-09-29T23:11:13.110 INFO:teuthology.orchestra.run.smithi012.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 11 MB/s | 225 kB 00:00 2023-09-29T23:11:13.186 INFO:teuthology.orchestra.run.smithi012.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2023-09-29T23:11:13.284 INFO:teuthology.orchestra.run.smithi012.stdout:(5/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 48 MB/s | 4.7 MB 00:00 2023-09-29T23:11:13.417 INFO:teuthology.orchestra.run.smithi012.stdout:(6/8): python3-scipy-1.0.0-20.module+el8.1.0+33 44 MB/s | 15 MB 00:00 2023-09-29T23:11:13.462 INFO:teuthology.orchestra.run.smithi012.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 26 MB/s | 4.6 MB 00:00 2023-09-29T23:11:13.542 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-29T23:11:13.641 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:13.642 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:13.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T23:11:14.152 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:14.940 INFO:teuthology.orchestra.run.smithi012.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 3.9 MB/s | 7.4 MB 00:01 2023-09-29T23:11:14.942 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:14.942 INFO:teuthology.orchestra.run.smithi012.stdout:Total 19 MB/s | 36 MB 00:01 2023-09-29T23:11:14.953 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:11:14.977 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:11:14.977 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:11:14.989 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:15.419 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:11:15.423 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:11:16.031 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:11:16.249 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-29T23:11:16.392 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-29T23:11:16.557 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-29T23:11:16.787 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:16.788 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================================================== 2023-09-29T23:11:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================================================== 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-29T23:11:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================================================== 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout:Install 8 Packages 2023-09-29T23:11:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:16.791 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 36 M 2023-09-29T23:11:16.791 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 199 M 2023-09-29T23:11:16.791 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:11:16.837 INFO:teuthology.orchestra.run.smithi153.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 4.7 MB/s | 225 kB 00:00 2023-09-29T23:11:16.905 INFO:teuthology.orchestra.run.smithi153.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 56 MB/s | 3.7 MB 00:00 2023-09-29T23:11:16.989 INFO:teuthology.orchestra.run.smithi153.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 57 MB/s | 4.7 MB 00:00 2023-09-29T23:11:17.132 INFO:teuthology.orchestra.run.smithi153.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 43 MB/s | 15 MB 00:00 2023-09-29T23:11:17.141 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-29T23:11:17.151 INFO:teuthology.orchestra.run.smithi153.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 9.0 MB/s | 169 kB 00:00 2023-09-29T23:11:17.175 INFO:teuthology.orchestra.run.smithi153.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-09-29T23:11:17.189 INFO:teuthology.orchestra.run.smithi153.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 23 MB/s | 4.6 MB 00:00 2023-09-29T23:11:17.332 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-29T23:11:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-29T23:11:18.140 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-29T23:11:18.272 INFO:teuthology.orchestra.run.smithi153.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 5.0 MB/s | 7.4 MB 00:01 2023-09-29T23:11:18.274 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:18.274 INFO:teuthology.orchestra.run.smithi153.stdout:Total 24 MB/s | 36 MB 00:01 2023-09-29T23:11:18.284 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:11:18.304 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:11:18.304 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:11:18.712 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:11:18.716 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:11:19.074 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-29T23:11:19.377 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:11:19.414 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-29T23:11:19.589 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-29T23:11:19.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-29T23:11:19.732 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-29T23:11:19.880 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-29T23:11:20.464 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-29T23:11:20.621 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-29T23:11:21.158 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-29T23:11:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-29T23:11:21.832 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-29T23:11:22.240 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-29T23:11:22.598 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-29T23:11:22.813 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-29T23:11:23.131 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-29T23:11:23.290 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-09-29T23:11:23.602 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-09-29T23:11:23.603 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-09-29T23:11:23.603 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-09-29T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-09-29T23:11:24.601 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:24.897 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-29T23:11:26.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-29T23:11:26.371 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:11:26.671 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-29T23:11:26.672 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-29T23:11:26.674 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-29T23:11:26.782 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-29T23:11:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-29T23:11:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-29T23:11:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-29T23:11:26.783 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:26.783 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:26.920 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-rook 2023-09-29T23:11:27.308 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:27.692 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-29T23:11:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:28.109 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:29.557 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:11:29.658 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:29.658 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:11:29.658 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:29.658 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-29T23:11:29.658 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:29.659 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:29.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2023-09-29T23:11:30.083 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:30.085 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================== 2023-09-29T23:11:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:11:30.085 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================== 2023-09-29T23:11:30.085 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:11:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-29T23:11:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================================== 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout:Install 13 Packages 2023-09-29T23:11:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:30.088 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 1.9 M 2023-09-29T23:11:30.088 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 25 M 2023-09-29T23:11:30.088 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:11:30.167 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:30.173 INFO:teuthology.orchestra.run.smithi012.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 303 kB/s | 16 kB 00:00 2023-09-29T23:11:30.181 INFO:teuthology.orchestra.run.smithi012.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 539 kB/s | 33 kB 00:00 2023-09-29T23:11:30.219 INFO:teuthology.orchestra.run.smithi012.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.7 MB/s | 81 kB 00:00 2023-09-29T23:11:30.244 INFO:teuthology.orchestra.run.smithi012.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 2.5 MB/s | 64 kB 00:00 2023-09-29T23:11:30.252 INFO:teuthology.orchestra.run.smithi012.stdout:(5/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 356 kB/s | 47 kB 00:00 2023-09-29T23:11:30.275 INFO:teuthology.orchestra.run.smithi012.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 1.9 MB/s | 61 kB 00:00 2023-09-29T23:11:30.288 INFO:teuthology.orchestra.run.smithi012.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 11 MB/s | 1.2 MB 00:00 2023-09-29T23:11:30.320 INFO:teuthology.orchestra.run.smithi012.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.2 MB/s | 155 kB 00:00 2023-09-29T23:11:30.332 INFO:teuthology.orchestra.run.smithi012.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 749 kB/s | 43 kB 00:00 2023-09-29T23:11:30.349 INFO:teuthology.orchestra.run.smithi012.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 498 kB/s | 27 kB 00:00 2023-09-29T23:11:30.350 INFO:teuthology.orchestra.run.smithi012.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 6.1 MB/s | 110 kB 00:00 2023-09-29T23:11:30.355 INFO:teuthology.orchestra.run.smithi012.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.6 MB/s | 126 kB 00:00 2023-09-29T23:11:30.365 INFO:teuthology.orchestra.run.smithi012.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 1.3 MB/s | 20 kB 00:00 2023-09-29T23:11:30.365 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:30.365 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.9 MB/s | 1.9 MB 00:00 2023-09-29T23:11:30.371 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:11:30.386 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:11:30.386 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:11:30.476 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:11:30.481 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:11:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:11:31.043 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-29T23:11:31.556 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-29T23:11:31.954 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-09-29T23:11:32.210 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-09-29T23:11:32.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-29T23:11:32.719 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-29T23:11:32.866 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:32.868 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================== 2023-09-29T23:11:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:11:32.868 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================== 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-29T23:11:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================================== 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2023-09-29T23:11:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:32.871 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.9 M 2023-09-29T23:11:32.871 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 25 M 2023-09-29T23:11:32.871 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:11:32.946 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-29T23:11:33.034 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 361 kB/s | 47 kB 00:00 2023-09-29T23:11:33.109 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 79 kB/s | 16 kB 00:00 2023-09-29T23:11:33.127 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 147 kB/s | 33 kB 00:00 2023-09-29T23:11:33.130 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-29T23:11:33.230 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 626 kB/s | 64 kB 00:00 2023-09-29T23:11:33.287 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 321 kB/s | 81 kB 00:00 2023-09-29T23:11:33.322 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 669 kB/s | 61 kB 00:00 2023-09-29T23:11:33.323 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 750 kB/s | 27 kB 00:00 2023-09-29T23:11:33.342 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 5.6 MB/s | 110 kB 00:00 2023-09-29T23:11:33.358 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 1.2 MB/s | 20 kB 00:00 2023-09-29T23:11:33.360 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.3 MB/s | 126 kB 00:00 2023-09-29T23:11:33.374 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 10 MB/s | 155 kB 00:00 2023-09-29T23:11:33.377 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 2.5 MB/s | 43 kB 00:00 2023-09-29T23:11:33.433 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.6 MB/s | 1.2 MB 00:00 2023-09-29T23:11:33.436 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:33.436 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.4 MB/s | 1.9 MB 00:00 2023-09-29T23:11:33.437 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-29T23:11:33.442 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:11:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:11:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:11:33.545 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:11:33.549 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:11:34.097 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:11:34.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-29T23:11:34.654 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-29T23:11:34.660 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-29T23:11:34.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-29T23:11:34.929 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-29T23:11:35.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:11:35.218 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-29T23:11:35.347 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:11:35.347 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-09-29T23:11:35.348 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-09-29T23:11:35.493 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-29T23:11:35.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-29T23:11:35.968 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-29T23:11:36.245 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-29T23:11:36.363 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-09-29T23:11:36.364 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:36.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-29T23:11:36.804 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-29T23:11:37.994 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-29T23:11:38.330 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-29T23:11:38.361 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:11:38.471 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:38.471 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:11:38.471 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:38.471 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:38.472 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:38.494 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:11:38.607 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T23:11:38.759 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:11:38.759 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-29T23:11:38.759 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-29T23:11:38.759 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-29T23:11:38.759 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-29T23:11:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-29T23:11:38.999 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:39.772 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:39.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-29T23:11:39.821 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:41.674 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================================== 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================================== 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:11:41.675 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================================== 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout:Install 5 Packages 2023-09-29T23:11:41.676 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 6.0 M 2023-09-29T23:11:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 26 M 2023-09-29T23:11:41.677 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:11:41.731 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:11:41.830 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-29T23:11:41.831 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-29T23:11:41.832 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-29T23:11:41.832 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-29T23:11:41.832 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-29T23:11:41.832 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:41.832 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:41.885 INFO:teuthology.orchestra.run.smithi012.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 799 kB/s | 121 kB 00:00 2023-09-29T23:11:41.896 INFO:teuthology.orchestra.run.smithi012.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 371 kB/s | 60 kB 00:00 2023-09-29T23:11:41.942 INFO:teuthology.orchestra.run.smithi012.stdout:(3/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2023-09-29T23:11:41.986 INFO:teuthology.orchestra.run.smithi012.stdout:(4/5): python3-babel-2.5.1-5.el8.noarch.rpm 47 MB/s | 4.8 MB 00:00 2023-09-29T23:11:42.004 INFO:teuthology.orchestra.run.smithi012.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.7 MB/s | 476 kB 00:00 2023-09-29T23:11:42.004 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:42.005 INFO:teuthology.orchestra.run.smithi012.stdout:Total 18 MB/s | 6.0 MB 00:00 2023-09-29T23:11:42.008 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:11:42.019 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:11:42.019 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:11:42.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T23:11:42.121 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:11:42.126 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:11:42.420 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:43.185 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:43.558 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-29T23:11:43.807 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-29T23:11:44.127 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-29T23:11:44.416 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-29T23:11:44.584 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:11:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:11:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-29T23:11:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-29T23:11:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-29T23:11:44.864 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-29T23:11:45.139 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================================== 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================================== 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-29T23:11:45.140 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================================== 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout:Install 5 Packages 2023-09-29T23:11:45.141 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:45.142 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.0 M 2023-09-29T23:11:45.142 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2023-09-29T23:11:45.142 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:11:45.527 INFO:teuthology.orchestra.run.smithi153.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 838 kB/s | 60 kB 00:00 2023-09-29T23:11:45.609 INFO:teuthology.orchestra.run.smithi153.stdout:(2/5): python3-babel-2.5.1-5.el8.noarch.rpm 58 MB/s | 4.8 MB 00:00 2023-09-29T23:11:45.610 INFO:teuthology.orchestra.run.smithi153.stdout:(3/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 778 kB/s | 121 kB 00:00 2023-09-29T23:11:45.626 INFO:teuthology.orchestra.run.smithi153.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 30 MB/s | 538 kB 00:00 2023-09-29T23:11:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2023-09-29T23:11:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 6.0 MB 00:00 2023-09-29T23:11:45.690 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:11:45.700 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:11:45.700 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:11:45.795 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:11:45.799 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:11:45.869 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-29T23:11:45.870 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:46.838 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:11:47.242 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-29T23:11:47.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-29T23:11:47.832 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:11:47.902 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-29T23:11:47.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:47.939 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:47.940 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:48.059 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-fuse 2023-09-29T23:11:48.181 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-29T23:11:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:11:48.453 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:48.617 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:11:48.618 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-29T23:11:48.618 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-29T23:11:48.618 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-29T23:11:48.618 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-29T23:11:49.222 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:49.626 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-29T23:11:49.627 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:51.549 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:11:51.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:51.650 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:11:51.734 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:11:51.734 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:11:51.735 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:11:51.736 INFO:teuthology.orchestra.run.smithi012.stdout:Install 2 Packages 2023-09-29T23:11:51.736 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:51.736 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 957 k 2023-09-29T23:11:51.736 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 3.0 M 2023-09-29T23:11:51.736 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:11:51.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2023-09-29T23:11:51.798 INFO:teuthology.orchestra.run.smithi012.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.3 MB/s | 83 kB 00:00 2023-09-29T23:11:51.952 INFO:teuthology.orchestra.run.smithi012.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 4.0 MB/s | 874 kB 00:00 2023-09-29T23:11:51.953 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:51.953 INFO:teuthology.orchestra.run.smithi012.stdout:Total 4.3 MB/s | 957 kB 00:00 2023-09-29T23:11:51.955 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:11:51.977 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:11:51.978 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:11:52.066 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:11:52.071 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:11:52.140 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:52.246 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:11:52.567 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-29T23:11:52.742 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:11:53.039 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:53.151 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:11:53.151 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-29T23:11:54.177 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-29T23:11:54.177 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:54.912 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:11:54.912 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2023-09-29T23:11:54.913 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-29T23:11:54.914 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:54.914 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:11:54.914 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:11:54.914 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2023-09-29T23:11:54.914 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:11:54.917 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 957 k 2023-09-29T23:11:54.917 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 3.0 M 2023-09-29T23:11:54.917 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:11:54.946 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 83 kB 00:00 2023-09-29T23:11:56.072 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 756 kB/s | 874 kB 00:01 2023-09-29T23:11:56.073 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:11:56.073 INFO:teuthology.orchestra.run.smithi153.stdout:Total 825 kB/s | 957 kB 00:01 2023-09-29T23:11:56.075 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:11:56.097 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:11:56.098 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:11:56.187 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:11:56.188 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:11:56.191 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:11:56.297 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:56.298 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:11:56.298 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-29T23:11:56.298 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:11:56.298 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:11:56.366 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:11:56.678 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-29T23:11:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:11:56.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-volume 2023-09-29T23:11:57.257 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:11:57.257 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-29T23:11:57.346 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:11:58.239 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:54 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:11:58.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-29T23:11:58.314 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:54 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:00.156 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:00.157 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:12:00.158 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:00.158 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 267 k 2023-09-29T23:12:00.158 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 1.3 M 2023-09-29T23:12:00.158 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:12:00.285 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:12:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:00.382 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:12:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-29T23:12:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:00.383 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:00.452 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 908 kB/s | 267 kB 00:00 2023-09-29T23:12:00.453 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:00.453 INFO:teuthology.orchestra.run.smithi012.stdout:Total 903 kB/s | 267 kB 00:00 2023-09-29T23:12:00.453 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:12:00.465 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:12:00.465 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:12:00.493 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:12:00.498 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:12:00.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2023-09-29T23:12:00.690 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:12:00.903 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:00.956 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:01.392 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:01.824 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:57 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:02.375 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:02.375 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:02:58 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:03.735 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:03.735 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:03.735 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:12:03.736 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:03.737 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 267 k 2023-09-29T23:12:03.737 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.3 M 2023-09-29T23:12:03.737 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:12:03.910 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-29T23:12:03.913 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:03.914 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-29T23:12:03.914 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:12:03.922 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:12:03.922 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:12:03.945 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:12:03.950 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:12:04.155 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:12:04.371 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:12:04.431 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:04.477 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:04.477 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:12:04.477 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:12:04.477 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:04.477 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:04.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados-devel 2023-09-29T23:12:04.853 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:04.991 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:05.829 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:05.895 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:12:05.895 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:12:07.736 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 130 k 2023-09-29T23:12:07.737 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 464 k 2023-09-29T23:12:07.738 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:12:07.844 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:12:07.941 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:12:07.941 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:12:07.941 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:07.941 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:08.079 INFO:teuthology.orchestra.run.smithi012.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 380 kB/s | 130 kB 00:00 2023-09-29T23:12:08.080 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:08.081 INFO:teuthology.orchestra.run.smithi012.stdout:Total 377 kB/s | 130 kB 00:00 2023-09-29T23:12:08.081 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:12:08.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2023-09-29T23:12:08.089 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:12:08.089 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:12:08.149 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:12:08.154 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:12:08.426 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:12:08.468 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:08.620 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:08.903 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:09.340 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:09.893 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:09.893 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:11.134 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:11.135 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:11.135 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-09-29T23:12:11.135 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:11.135 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:12:11.135 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 130 k 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 464 k 2023-09-29T23:12:11.136 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:12:11.288 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 858 kB/s | 130 kB 00:00 2023-09-29T23:12:11.288 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:11.289 INFO:teuthology.orchestra.run.smithi153.stdout:Total 850 kB/s | 130 kB 00:00 2023-09-29T23:12:11.289 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:12:11.295 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:12:11.295 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:12:11.351 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:12:11.355 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:12:11.480 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:12:11.679 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:11.834 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:12:11.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:11.940 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:12:11.940 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:11.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:11.940 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:11.942 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:12.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs2 2023-09-29T23:12:12.455 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:12.941 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:12.941 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:08 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:13.340 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:09 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:14.737 INFO:teuthology.orchestra.run.smithi012.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:14.824 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:12:14.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:14.922 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:12:14.922 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:14.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:14.922 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:15.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2023-09-29T23:12:15.233 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:15.234 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:15.234 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:15.337 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs-devel 2023-09-29T23:12:15.419 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:15.730 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:16.320 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:12 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:16.579 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:12 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:17.684 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:18.155 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:18.156 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:18.156 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:18.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2023-09-29T23:12:18.483 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:18.483 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:18.483 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-09-29T23:12:18.483 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:18.483 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:12:18.484 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:18.485 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 25 k 2023-09-29T23:12:18.485 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 109 k 2023-09-29T23:12:18.485 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:12:18.596 INFO:teuthology.orchestra.run.smithi012.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 223 kB/s | 25 kB 00:00 2023-09-29T23:12:18.596 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:18.597 INFO:teuthology.orchestra.run.smithi012.stdout:Total 220 kB/s | 25 kB 00:00 2023-09-29T23:12:18.597 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:12:18.601 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:12:18.601 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:12:18.603 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:12:18.607 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:12:18.645 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:18.663 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:12:18.810 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:19.054 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:19.538 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:15 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:20.039 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:20.039 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:16 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:12:21.620 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:21.621 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2023-09-29T23:12:21.622 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 109 k 2023-09-29T23:12:21.622 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:12:22.017 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:12:22.117 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 50 kB/s | 25 kB 00:00 2023-09-29T23:12:22.118 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:22.118 INFO:teuthology.orchestra.run.smithi153.stdout:Total 50 kB/s | 25 kB 00:00 2023-09-29T23:12:22.118 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:12:22.122 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:12:22.122 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:12:22.124 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:12:22.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:22.126 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:12:22.126 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:22.126 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:22.126 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:22.129 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:12:22.190 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:12:22.250 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados2 2023-09-29T23:12:22.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:22.600 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:23.447 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:19 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:23.614 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:23.614 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:19 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:24.935 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:25.433 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:25.435 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:25.435 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:25.535 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librbd1 2023-09-29T23:12:25.569 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:12:25.666 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:25.666 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:12:25.666 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:25.666 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:25.667 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:25.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2023-09-29T23:12:25.929 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:26.151 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:26.749 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:22 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:27.028 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:23 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:28.217 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:28.469 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:28.709 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:28.711 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:28.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rados 2023-09-29T23:12:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:28.946 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:28.946 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:29.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2023-09-29T23:12:29.211 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:29.428 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:30.080 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:26 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:30.316 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:26 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:31.550 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:31.790 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:32.080 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:32.081 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:32.081 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:32.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rgw 2023-09-29T23:12:32.262 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:32.263 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:32.263 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:32.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2023-09-29T23:12:32.581 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:32.745 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:33.379 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:29 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:33.573 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:29 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:34.881 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:34.907 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:35.355 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:35.356 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:35.356 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:35.399 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:35.400 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:35.401 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:35.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2023-09-29T23:12:35.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-cephfs 2023-09-29T23:12:35.830 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:35.893 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:36.660 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:32 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:36.673 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:32 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:38.014 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:38.040 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:38.485 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:38.486 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:38.486 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:38.553 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:38.553 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:38.554 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:38.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2023-09-29T23:12:38.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rbd 2023-09-29T23:12:38.966 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:39.045 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:39.815 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:35 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:39.844 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:35 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:41.230 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:41.289 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:41.701 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:41.702 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:41.703 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:41.789 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:41.790 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2023-09-29T23:12:41.790 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:41.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2023-09-29T23:12:41.890 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-fuse 2023-09-29T23:12:42.210 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:42.284 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:39 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:43.154 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:39 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:44.470 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:12:44.941 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:44.942 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2023-09-29T23:12:44.942 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:45.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2023-09-29T23:12:45.126 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:45.127 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:12:45.128 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:45.128 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 90 k 2023-09-29T23:12:45.128 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 264 k 2023-09-29T23:12:45.128 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:12:45.416 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:45.673 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 166 kB/s | 90 kB 00:00 2023-09-29T23:12:45.674 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:45.674 INFO:teuthology.orchestra.run.smithi012.stdout:Total 165 kB/s | 90 kB 00:00 2023-09-29T23:12:45.674 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:12:45.687 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:12:45.688 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:12:45.749 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:12:45.753 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:12:45.869 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:12:46.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:46.331 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:46.333 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:42 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:47.280 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:47.280 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:43 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:48.147 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:48.147 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:12:48.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:48.149 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 90 k 2023-09-29T23:12:48.149 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 264 k 2023-09-29T23:12:48.149 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:12:48.558 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 221 kB/s | 90 kB 00:00 2023-09-29T23:12:48.558 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:48.558 INFO:teuthology.orchestra.run.smithi153.stdout:Total 220 kB/s | 90 kB 00:00 2023-09-29T23:12:48.559 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:12:48.570 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:12:48.570 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:12:48.625 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:12:48.629 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:12:48.767 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:12:48.973 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:49.340 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:12:49.447 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:49.448 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:12:49.448 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:49.448 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:49.448 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:49.570 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-mirror 2023-09-29T23:12:49.965 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:50.264 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:50.264 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:46 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:50.814 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:46 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:52.149 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:12:52.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:52.246 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:12:52.246 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:52.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:52.246 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:12:52.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2023-09-29T23:12:52.703 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:12:52.704 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 3.5 M 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 16 M 2023-09-29T23:12:52.705 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:12:52.736 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:53.676 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:49 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:53.778 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 3.2 MB/s | 3.5 MB 00:01 2023-09-29T23:12:53.779 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:53.779 INFO:teuthology.orchestra.run.smithi012.stdout:Total 3.2 MB/s | 3.5 MB 00:01 2023-09-29T23:12:53.780 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:12:53.795 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:12:53.795 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:12:53.870 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:12:53.875 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:12:54.360 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:12:54.553 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:54.989 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:12:55.558 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:12:55.559 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.5 M 2023-09-29T23:12:55.560 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 16 M 2023-09-29T23:12:55.560 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:12:55.941 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 9.0 MB/s | 3.5 MB 00:00 2023-09-29T23:12:55.942 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:12:55.943 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.0 MB/s | 3.5 MB 00:00 2023-09-29T23:12:55.943 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:12:55.956 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:12:55.957 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:12:55.992 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:55.992 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:51 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:56.024 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:12:56.028 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:12:56.491 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:12:56.657 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:57.102 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:12:58.157 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:54 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:12:58.244 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:12:58.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:58.348 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:12:58.348 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:12:58.348 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:12:58.348 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:12:58.467 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-nbd 2023-09-29T23:12:58.868 INFO:teuthology.orchestra.run.smithi012.stdout:Updating Subscription Management repositories. 2023-09-29T23:12:59.770 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:55 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:13:00.181 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:13:00.280 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:00.280 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:13:00.280 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:13:00.280 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:00.280 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:13:00.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2023-09-29T23:13:00.775 INFO:teuthology.orchestra.run.smithi153.stdout:Updating Subscription Management repositories. 2023-09-29T23:13:01.626 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:57 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:13:01.756 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-09-29T23:13:01.757 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:13:01.757 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-09-29T23:13:01.757 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:13:01.757 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-09-29T23:13:01.757 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 173 k 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 552 k 2023-09-29T23:13:01.758 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-09-29T23:13:02.016 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 673 kB/s | 173 kB 00:00 2023-09-29T23:13:02.016 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:13:02.016 INFO:teuthology.orchestra.run.smithi012.stdout:Total 669 kB/s | 173 kB 00:00 2023-09-29T23:13:02.017 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-09-29T23:13:02.031 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-09-29T23:13:02.031 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-09-29T23:13:02.091 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-09-29T23:13:02.095 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-09-29T23:13:02.219 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-09-29T23:13:02.408 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:02.687 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2023-09-29T23:13:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2023-09-29T23:13:03.410 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 173 k 2023-09-29T23:13:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 552 k 2023-09-29T23:13:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2023-09-29T23:13:03.616 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 845 kB/s | 173 kB 00:00 2023-09-29T23:13:03.616 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:13:03.617 INFO:teuthology.orchestra.run.smithi153.stdout:Total 839 kB/s | 173 kB 00:00 2023-09-29T23:13:03.617 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2023-09-29T23:13:03.622 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:03.622 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:03:59 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:13:03.630 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2023-09-29T23:13:03.630 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2023-09-29T23:13:03.686 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2023-09-29T23:13:03.691 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2023-09-29T23:13:03.843 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2023-09-29T23:13:04.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:04.340 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:05.391 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:13:05.391 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:01 ago on Fri 29 Sep 2023 11:09:04 PM UTC. 2023-09-29T23:13:05.735 INFO:teuthology.orchestra.run.smithi012.stdout:Installed products updated. 2023-09-29T23:13:05.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:05.842 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-09-29T23:13:05.843 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:13:05.843 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:05.843 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-09-29T23:13:05.994 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:07.425 INFO:teuthology.orchestra.run.smithi153.stdout:Installed products updated. 2023-09-29T23:13:07.522 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:07.522 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2023-09-29T23:13:07.522 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:13:07.522 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:07.522 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2023-09-29T23:13:07.640 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:07.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:13:07.866 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T23:13:07.891 INFO:teuthology.orchestra.run.smithi012.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-29T23:13:07.891 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-29T23:13:07.891 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-29T23:13:07.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:13:08.036 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T23:13:08.056 INFO:teuthology.orchestra.run.smithi153.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-29T23:13:08.056 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-29T23:13:08.056 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-29T23:13:08.057 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-29T23:13:08.057 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:08.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T23:13:08.089 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:08.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T23:13:08.122 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-29T23:13:08.122 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:08.122 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T23:13:08.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T23:13:08.226 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:08.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T23:13:08.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T23:13:08.314 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-29T23:13:08.314 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:08.314 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T23:13:08.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T23:13:08.408 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:08.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T23:13:08.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T23:13:08.496 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-29T23:13:08.497 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:08.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T23:13:08.523 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T23:13:08.590 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:08.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T23:13:08.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T23:13:08.678 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T23:13:08.742 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-29T23:13:08.742 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2023-09-29T23:13:08.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2023-09-29T23:13:08.769 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-29T23:13:08.769 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T23:13:08.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T23:13:08.837 INFO:tasks.ceph:Creating extra log directories... 2023-09-29T23:13:08.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T23:13:08.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T23:13:08.915 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-29T23:13:08.915 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T23:13:08.916 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406532', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCXEjO1dqSmbvU7ggLljVXfP71JORi5hJrFaY3B58mHbBN58u4n1UxhMi8w6e3hO+p5ubHQ/W04fNy2DD5Xew=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEM5Ub7/Hqh7BEtkSEpSolDqKk4UjAfv947YCtF6f6+zrXTNFwNmC2cxGfrRNbRbOoq17u1in5gC8LzFf0TVXvs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T23:13:08.916 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-29T23:13:08.947 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-29T23:13:08.962 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T23:13:09.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T23:13:09.028 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:09.028 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T23:13:09.082 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T23:13:09.083 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 219599 Links: 1 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-09-29 23:12:54.750436209 +0000 2023-09-29T23:13:09.139 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-09-29 23:08:17.742714440 +0000 2023-09-29T23:13:09.140 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-09-29 23:08:17.742714440 +0000 2023-09-29T23:13:09.140 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2023-09-29T23:13:09.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T23:13:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-09-29T23:13:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-09-29T23:13:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000250634 s, 2.0 MB/s 2023-09-29T23:13:09.204 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T23:13:09.261 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2023-09-29T23:13:09.316 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 219665 Links: 1 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-09-29 23:12:54.750436209 +0000 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-09-29 23:08:18.241711294 +0000 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-09-29 23:08:18.241711294 +0000 2023-09-29T23:13:09.317 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2023-09-29T23:13:09.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T23:13:09.380 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-09-29T23:13:09.381 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-09-29T23:13:09.381 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000226688 s, 2.3 MB/s 2023-09-29T23:13:09.382 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T23:13:09.438 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2023-09-29T23:13:09.493 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T23:13:09.493 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:09.493 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 219700 Links: 1 2023-09-29T23:13:09.493 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:09.493 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:09.494 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-09-29 23:12:54.750436209 +0000 2023-09-29T23:13:09.494 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-09-29 23:08:18.744708123 +0000 2023-09-29T23:13:09.494 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-09-29 23:08:18.744708123 +0000 2023-09-29T23:13:09.494 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2023-09-29T23:13:09.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T23:13:09.556 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-09-29T23:13:09.556 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-09-29T23:13:09.557 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000240069 s, 2.1 MB/s 2023-09-29T23:13:09.557 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T23:13:09.614 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2023-09-29T23:13:09.669 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T23:13:09.669 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:09.669 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 219843 Links: 1 2023-09-29T23:13:09.669 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:09.670 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:09.670 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-09-29 23:12:54.750436209 +0000 2023-09-29T23:13:09.670 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-09-29 23:08:19.226705085 +0000 2023-09-29T23:13:09.670 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-09-29 23:08:19.226705085 +0000 2023-09-29T23:13:09.670 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2023-09-29T23:13:09.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T23:13:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-09-29T23:13:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-09-29T23:13:09.734 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00019158 s, 2.7 MB/s 2023-09-29T23:13:09.735 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T23:13:09.792 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:09.792 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:09.792 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T23:13:09.807 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T23:13:09.807 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2023-09-29T23:13:09.862 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T23:13:09.862 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:09.862 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 219650 Links: 1 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-09-29 23:12:56.869282082 +0000 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-09-29 23:08:03.152319943 +0000 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-09-29 23:08:03.152319943 +0000 2023-09-29T23:13:09.863 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-09-29T23:13:09.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T23:13:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-09-29T23:13:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-09-29T23:13:09.925 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000251116 s, 2.0 MB/s 2023-09-29T23:13:09.926 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T23:13:09.982 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 221499 Links: 1 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-09-29 23:12:56.869282082 +0000 2023-09-29T23:13:10.038 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-09-29 23:08:03.668310422 +0000 2023-09-29T23:13:10.039 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-09-29 23:08:03.668310422 +0000 2023-09-29T23:13:10.039 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-09-29T23:13:10.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T23:13:10.101 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-09-29T23:13:10.101 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-09-29T23:13:10.101 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000226138 s, 2.3 MB/s 2023-09-29T23:13:10.102 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T23:13:10.157 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2023-09-29T23:13:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T23:13:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:10.212 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 222441 Links: 1 2023-09-29T23:13:10.212 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:10.212 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:10.213 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-09-29 23:12:56.869282082 +0000 2023-09-29T23:13:10.213 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-09-29 23:08:04.177301034 +0000 2023-09-29T23:13:10.213 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-09-29 23:08:04.177301034 +0000 2023-09-29T23:13:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-09-29T23:13:10.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T23:13:10.275 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-09-29T23:13:10.275 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-09-29T23:13:10.275 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000221236 s, 2.3 MB/s 2023-09-29T23:13:10.276 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T23:13:10.331 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2023-09-29T23:13:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T23:13:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:13:10.387 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 6h/6d Inode: 218499 Links: 1 2023-09-29T23:13:10.387 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:13:10.387 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:13:10.388 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-09-29 23:12:56.869282082 +0000 2023-09-29T23:13:10.388 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-09-29 23:08:04.669291958 +0000 2023-09-29T23:13:10.388 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-09-29 23:08:04.669291958 +0000 2023-09-29T23:13:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-09-29T23:13:10.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T23:13:10.449 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-09-29T23:13:10.450 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-09-29T23:13:10.450 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000229187 s, 2.2 MB/s 2023-09-29T23:13:10.451 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T23:13:10.506 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:10.507 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-29T23:13:10.507 INFO:tasks.ceph:Generating config... 2023-09-29T23:13:10.508 INFO:tasks.ceph:[client] debug monc = 20 2023-09-29T23:13:10.508 INFO:tasks.ceph:[client] debug ms = 1 2023-09-29T23:13:10.508 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-29T23:13:10.508 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-29T23:13:10.508 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-29T23:13:10.508 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-29T23:13:10.509 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-29T23:13:10.509 INFO:tasks.ceph:[global] ms type = async 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-29T23:13:10.509 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-09-29T23:13:10.510 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-09-29T23:13:10.510 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-29T23:13:10.510 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd objectstore = filestore 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-29T23:13:10.510 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-29T23:13:10.511 INFO:tasks.ceph:[osd] osd sloppy crc = True 2023-09-29T23:13:10.511 INFO:tasks.ceph:Setting up mon.a... 2023-09-29T23:13:10.511 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:10.564 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2023-09-29T23:13:10.567 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-29T23:13:10.609 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:13:10.674 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.b', '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]'), ('mon.c', '[v2:172.21.15.12:3302,v1:172.21.15.12:6791]'), ('mon.d', '[v2:172.21.15.12:3303,v1:172.21.15.12:6792]'), ('mon.e', '[v2:172.21.15.12:3304,v1:172.21.15.12:6793]'), ('mon.f', '172.21.15.153'), ('mon.g', '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]'), ('mon.h', '[v2:172.21.15.153:3302,v1:172.21.15.153:6791]'), ('mon.i', '[v2:172.21.15.153:3303,v1:172.21.15.153:6792]')] 2023-09-29T23:13:10.675 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],[v2:172.21.15.12:3302,v1:172.21.15.12:6791],[v2:172.21.15.12:3303,v1:172.21.15.12:6792],[v2:172.21.15.12:3304,v1:172.21.15.12:6793],172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],[v2:172.21.15.153:3302,v1:172.21.15.153:6791],[v2:172.21.15.153:3303,v1:172.21.15.153:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2023-09-29T23:13:10.676 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:10.676 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T23:13:10.730 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --addv b '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]' --addv c '[v2:172.21.15.12:3302,v1:172.21.15.12:6791]' --addv d '[v2:172.21.15.12:3303,v1:172.21.15.12:6792]' --addv e '[v2:172.21.15.12:3304,v1:172.21.15.12:6793]' --add f 172.21.15.153 --addv g '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]' --addv h '[v2:172.21.15.153:3302,v1:172.21.15.153:6791]' --addv i '[v2:172.21.15.153:3303,v1:172.21.15.153:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = octopus 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 15 (octopus) 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 3 2023-09-29T23:13:10.833 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:3: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:4: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:5: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:6: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:7: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:8: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:10.834 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2023-09-29T23:13:10.835 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T23:13:10.850 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b8fec2b7-0020-4de5-aeb3-24af8bd53167... 2023-09-29T23:13:10.851 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-29T23:13:10.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.934 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.935 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2023-09-29T23:13:10.936 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2023-09-29T23:13:10.937 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],[v2:172.21.15.12:3302,v1:172.21.15.12:6791],[v2:172.21.15.12:3303,v1:172.21.15.12:6792],[v2:172.21.15.12:3304,v1:172.21.15.12:6793],172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],[v2:172.21.15.153:3302,v1:172.21.15.153:6791],[v2:172.21.15.153:3303,v1:172.21.15.153:6792]" 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: mon election default strategy = 3 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: ms bind msgr1 = False 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: ms bind msgr2 = True 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 5000 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: ms type = async 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2023-09-29T23:13:10.938 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.939 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2023-09-29T23:13:10.940 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2023-09-29T23:13:10.941 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore zero block detection = True 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify cached snaps = True 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify missing on start = True 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock override recovery settings = True 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = filestore 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue = debug_random 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd op queue cut off = debug_random 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: osd sloppy crc = True 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2023-09-29T23:13:10.942 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2023-09-29T23:13:10.943 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T23:13:10.944 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon min osdmap epochs = 25 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune interval = 2 2023-09-29T23:13:10.945 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune min = 15 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon osdmap full prune txsize = 2 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub inject crc mismatch = 0.01 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub inject missing keys = 0.05 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: mon scrub interval = 300 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: paxos service trim min = 5 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2023-09-29T23:13:10.946 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug monc = 20 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.d] 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.e] 2023-09-29T23:13:10.947 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.f] 2023-09-29T23:13:10.948 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.g] 2023-09-29T23:13:10.948 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.h] 2023-09-29T23:13:10.948 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.i] 2023-09-29T23:13:10.949 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2023-09-29T23:13:10.949 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2023-09-29T23:13:10.949 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T23:13:10.949 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2023-09-29T23:13:10.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2023-09-29T23:13:10.951 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.952 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],[v2:172.21.15.12:3302,v1:172.21.15.12:6791],[v2:172.21.15.12:3303,v1:172.21.15.12:6792],[v2:172.21.15.12:3304,v1:172.21.15.12:6793],172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],[v2:172.21.15.153:3302,v1:172.21.15.153:6791],[v2:172.21.15.153:3303,v1:172.21.15.153:6792]" 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 3 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr1 = False 2023-09-29T23:13:10.953 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = True 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2023-09-29T23:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2023-09-29T23:13:10.955 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2023-09-29T23:13:10.956 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = filestore 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd sloppy crc = True 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-09-29T23:13:10.957 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-09-29T23:13:10.958 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T23:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2023-09-29T23:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 25 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject crc mismatch = 0.01 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject missing keys = 0.05 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 5 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2023-09-29T23:13:10.961 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.d] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.e] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.f] 2023-09-29T23:13:10.962 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.g] 2023-09-29T23:13:10.963 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.h] 2023-09-29T23:13:10.963 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.i] 2023-09-29T23:13:10.963 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-29T23:13:10.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-29T23:13:11.042 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T23:13:11.042 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:11.042 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-29T23:13:11.057 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:11.058 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-29T23:13:11.113 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:13:11.113 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:11.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-29T23:13:11.113 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:13:11.188 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:11.188 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:11.243 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:13:11.243 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:11.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-29T23:13:11.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:13:11.279 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:11.279 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:11.334 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T23:13:11.334 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T23:13:11.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T23:13:11.449 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T23:13:11.451 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T23:13:11.451 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T23:13:11.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-29T23:13:11.489 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-29T23:13:11.501 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T23:13:11.502 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-29T23:13:11.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-29T23:13:11.527 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:11.527 INFO:tasks.ceph:role: osd.0 2023-09-29T23:13:11.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:13:11.529 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T23:13:11.601 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:11.601 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:11.601 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:11.601 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 2023-09-29T23:13:11.601 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:11.602 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:11.602 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:11.602 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:11.602 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:11.602 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:12.031 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-09-29T23:13:12.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-09-29T23:13:12.032 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-29T23:13:12.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-29T23:13:12.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-29T23:13:12.298 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:12.299 INFO:tasks.ceph:role: osd.1 2023-09-29T23:13:12.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:13:12.299 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T23:13:12.371 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:12.371 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:12.371 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:12.371 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:12.372 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:12.791 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-09-29T23:13:12.793 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-09-29T23:13:12.793 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-29T23:13:12.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-29T23:13:12.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-29T23:13:13.001 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:13.001 INFO:tasks.ceph:role: osd.2 2023-09-29T23:13:13.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:13:13.002 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T23:13:13.075 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:13.075 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:13.075 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:13.075 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:13.076 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:13.522 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-09-29T23:13:13.524 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-09-29T23:13:13.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-29T23:13:13.685 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-29T23:13:13.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:13.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:13.814+0000 7f8032e2b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-29T23:13:13.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:13.817+0000 7f8032e2b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-29T23:13:13.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:13.843+0000 7f8032e2b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:13.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:13.846+0000 7f8032e2b540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:14.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.122+0000 7f8032e2b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.127+0000 7f8032e2b540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.127+0000 7f8032e2b540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.152 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-29T23:13:14.179 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:14.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.262+0000 7f6a5f879540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-29T23:13:14.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.263+0000 7f6a5f879540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-29T23:13:14.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.292+0000 7f6a5f879540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:14.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.295+0000 7f6a5f879540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.540+0000 7f6a5f879540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.541+0000 7f6a5f879540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.541+0000 7f6a5f879540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.568 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-29T23:13:14.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:14.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.715+0000 7f8c8661d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-29T23:13:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.715+0000 7f8c8661d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-29T23:13:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.722+0000 7f8c8661d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.722+0000 7f8c8661d540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.879+0000 7f8c8661d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.879+0000 7f8c8661d540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:14.879+0000 7f8c8661d540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:14.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-29T23:13:14.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-29T23:13:14.982 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:14.982 INFO:tasks.ceph:role: osd.3 2023-09-29T23:13:14.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:13:14.983 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:15.057 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:15.058 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:15.058 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:15.058 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:15.058 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:15.493 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-09-29T23:13:15.495 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-09-29T23:13:15.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-09-29T23:13:15.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-29T23:13:15.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-29T23:13:15.767 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:15.767 INFO:tasks.ceph:role: osd.4 2023-09-29T23:13:15.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:13:15.767 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T23:13:15.840 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:15.841 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:16.271 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-09-29T23:13:16.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-09-29T23:13:16.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-09-29T23:13:16.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-29T23:13:16.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-29T23:13:16.486 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-29T23:13:16.486 INFO:tasks.ceph:role: osd.5 2023-09-29T23:13:16.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:13:16.487 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T23:13:16.558 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:13:16.559 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:13:17.002 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-09-29T23:13:17.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-09-29T23:13:17.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-09-29T23:13:17.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-29T23:13:17.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:17.321 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.321+0000 7fbf30493540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-29T23:13:17.322 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.322+0000 7fbf30493540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-29T23:13:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.348+0000 7fbf30493540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.351+0000 7fbf30493540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:17.586 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.586+0000 7fbf30493540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.589+0000 7fbf30493540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.589+0000 7fbf30493540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:17.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-29T23:13:17.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:17.727 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.727+0000 7f36ea968540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-29T23:13:17.727 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.727+0000 7f36ea968540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-29T23:13:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.759+0000 7f36ea968540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.760+0000 7f36ea968540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:17.973 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.973+0000 7f36ea968540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:17.976 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.975+0000 7f36ea968540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:17.976 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:17.975+0000 7f36ea968540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:17.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-29T23:13:18.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:18.140 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.140+0000 7fb750c0e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-29T23:13:18.140 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.140+0000 7fb750c0e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-29T23:13:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.145+0000 7fb750c0e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.145+0000 7fb750c0e540 -1 journal FileJournal::_open_file : zeroing journal 2023-09-29T23:13:18.327 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.326+0000 7fb750c0e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:18.327 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.326+0000 7fb750c0e540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:18.327 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:18.326+0000 7fb750c0e540 -1 journal do_read_entry(4096): bad header magic 2023-09-29T23:13:18.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-29T23:13:18.399 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T23:13:18.399 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:18.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-29T23:13:18.429 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:18.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-29T23:13:18.499 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:18.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-29T23:13:18.567 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:18.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-29T23:13:18.591 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:18.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-29T23:13:18.655 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:18.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-29T23:13:18.719 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:18.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-29T23:13:18.783 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:18.783 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-29T23:13:18.838 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T23:13:18.838 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-29T23:13:18.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-29T23:13:18.881 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2023-09-29T23:13:18.881 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAJWhdlV4ybMBAAQNYjlMvGpdVnoHvxp6aLqw== 2023-09-29T23:13:18.881 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2023-09-29T23:13:18.881 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAKWhdlWByrDxAAOGEIoXamMP185gF21EA8wA== 2023-09-29T23:13:18.881 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAKWhdlGMSzKhAAlIOKAvq8yCCqa49G/vNYrw== 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAHWhdlbfjPGhAAiBlTC0VOgGKMltRjmoh4gA== 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQANWhdlq085ExAAMAuLSiyJWmlQlJoFJv2AVA== 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQANWhdlg6RnKxAAvPUH86D6xdKeuZ9bnF3NUg== 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAOWhdlPqVqCBAAks0eWEITkNNtwfbXZ1BYag== 2023-09-29T23:13:18.882 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2023-09-29T23:13:18.883 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAHWhdlSOQ4HRAAn22JvTkjqwv6vLgeE6ghNA== 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAJWhdlV4ybMBAAQNYjlMvGpdVnoHvxp6aLqw== 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAKWhdlWByrDxAAOGEIoXamMP185gF21EA8wA== 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAKWhdlGMSzKhAAlIOKAvq8yCCqa49G/vNYrw== 2023-09-29T23:13:18.901 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAHWhdlbfjPGhAAiBlTC0VOgGKMltRjmoh4gA== 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQANWhdlq085ExAAMAuLSiyJWmlQlJoFJv2AVA== 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQANWhdlg6RnKxAAvPUH86D6xdKeuZ9bnF3NUg== 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAOWhdlPqVqCBAAks0eWEITkNNtwfbXZ1BYag== 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2023-09-29T23:13:18.902 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAHWhdlSOQ4HRAAn22JvTkjqwv6vLgeE6ghNA== 2023-09-29T23:13:18.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:18.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:18.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:18.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.092 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:13:19.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:13:19.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.239 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:13:19.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:13:19.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:13:19.339 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T23:13:19.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-29T23:13:19.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:19.458 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-29T23:13:19.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-29T23:13:19.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:19.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-29T23:13:19.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-29T23:13:19.733 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:19.826 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-29T23:13:19.851 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2023-09-29T23:13:19.916 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.009 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2023-09-29T23:13:20.034 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2023-09-29T23:13:20.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2023-09-29T23:13:20.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2023-09-29T23:13:20.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2023-09-29T23:13:20.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2023-09-29T23:13:20.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2023-09-29T23:13:20.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2023-09-29T23:13:20.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2023-09-29T23:13:20.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2023-09-29T23:13:20.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:13:20.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2023-09-29T23:13:20.906 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:20.908 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:13:20.960 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-29T23:13:20.961 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-29T23:13:20.961 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-29T23:13:20.966 INFO:tasks.ceph.mon.a:Started 2023-09-29T23:13:20.966 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-29T23:13:20.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-29T23:13:20.968 INFO:tasks.ceph.mon.b:Started 2023-09-29T23:13:20.968 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-29T23:13:20.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-29T23:13:20.970 INFO:tasks.ceph.mon.c:Started 2023-09-29T23:13:20.971 INFO:tasks.ceph.mon.d:Restarting daemon 2023-09-29T23:13:20.971 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2023-09-29T23:13:20.973 INFO:tasks.ceph.mon.d:Started 2023-09-29T23:13:20.973 INFO:tasks.ceph.mon.e:Restarting daemon 2023-09-29T23:13:20.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2023-09-29T23:13:20.975 INFO:tasks.ceph.mon.e:Started 2023-09-29T23:13:20.975 INFO:tasks.ceph.mon.f:Restarting daemon 2023-09-29T23:13:20.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2023-09-29T23:13:21.003 INFO:tasks.ceph.mon.f:Started 2023-09-29T23:13:21.004 INFO:tasks.ceph.mon.g:Restarting daemon 2023-09-29T23:13:21.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2023-09-29T23:13:21.006 INFO:tasks.ceph.mon.g:Started 2023-09-29T23:13:21.006 INFO:tasks.ceph.mon.h:Restarting daemon 2023-09-29T23:13:21.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2023-09-29T23:13:21.008 INFO:tasks.ceph.mon.h:Started 2023-09-29T23:13:21.008 INFO:tasks.ceph.mon.i:Restarting daemon 2023-09-29T23:13:21.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2023-09-29T23:13:21.010 INFO:tasks.ceph.mon.i:Started 2023-09-29T23:13:21.010 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-29T23:13:21.010 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-29T23:13:21.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-29T23:13:21.012 INFO:tasks.ceph.mgr.x:Started 2023-09-29T23:13:21.013 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T23:13:21.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2023-09-29T23:13:21.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.158+0000 7ff462013700 10 monclient: get_monmap_and_config 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.158+0000 7ff462013700 10 monclient: build_initial_monmap 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.158+0000 7ff462013700 10 monclient: monmap: 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:21.159350+0000 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:21.159350+0000 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:21.159 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:21.160 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 Processor -- start 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 -- start start 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient: init 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient: _add_conns ranks=[5,7,1,6,8,4,9,3,2,0,10] 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): picked mon.noname-h con 0x7ff45c0a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:21.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): picked mon.noname-i con 0x7ff45c0a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff45c133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c068ed0 con 0x7ff45c133740 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09e350 con 0x7ff45c0a1f90 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09ef70 con 0x7ff45c0a2360 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.159+0000 7ff462013700 10 monclient(hunting): authenticate will time out at 1110.519783s 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45bfff700 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff45c133740 0x7ff45c133bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:43652/0 (socket says 172.21.15.12:43652) 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 learned_addr learned my addr 172.21.15.12:0/2985315052 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff45c0a1f90 auth_method 0 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff45c0a2360 auth_method 0 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:13:21.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.160+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:13:21.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff45c0a2360 auth_method 0 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff45c0a1f90 auth_method 0 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:13:21.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-29T23:13:21.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff45c0a2360 auth_method 0 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff45c0a1f90 auth_method 0 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:21.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-29T23:13:21.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:21.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-29T23:13:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff45c0a2360 auth_method 0 2023-09-29T23:13:22.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff45c0a1f90 auth_method 0 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:22.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-29T23:13:22.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:22.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-29T23:13:24.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): tick 2023-09-29T23:13:24.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 monclient(hunting): continuing hunt 2023-09-29T23:13:24.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): _reopen_session rank -1 2023-09-29T23:13:24.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 msgr2=0x7ff45c0a27b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:13:24.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff45c0a2360 0x7ff45c0a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 msgr2=0x7ff45c09f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff45c0a1f90 0x7ff45c09f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 >> v1:172.21.15.153:6789/0 conn(0x7ff45c133740 legacy=0x7ff45c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient: _add_conns ranks=[0,10,6,8,3,4,9,2] 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff43c0038c0 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:24.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): picked mon.noname-f con 0x7ff43c0074f0 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): picked mon.noname-d con 0x7ff43c00a010 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c04c960 con 0x7ff43c0038c0 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c09e350 con 0x7ff43c0074f0 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:24.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09ef70 con 0x7ff43c00a010 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.159+0000 7ff4597fa700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff43c00a010 auth_method 0 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:24.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-29T23:13:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff43c0074f0 auth_method 0 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:24.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.160+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:13:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff43c00a010 auth_method 0 2023-09-29T23:13:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff43c0074f0 auth_method 0 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-29T23:13:24.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.361+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-29T23:13:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff43c00a010 auth_method 0 2023-09-29T23:13:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff43c0074f0 auth_method 0 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:24.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:24.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:24.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:24.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:24.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:24.762+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-29T23:13:25.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:25.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff43c00a010 auth_method 0 2023-09-29T23:13:25.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:25.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:25.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff43c0074f0 auth_method 0 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:25.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:25.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:25.563+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.371+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63878178 0 0) 0x7ff45c04c960 con 0x7ff43c0038c0 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.371+0000 7ff45a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.371+0000 7ff45a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.371+0000 7ff45a7fc700 10 monclient(hunting): my global_id is 4104 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.371+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448003570 con 0x7ff43c0038c0 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446561873 0 0) 0x7ff448003570 con 0x7ff43c0038c0 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 msgr2=0x7ff43c00c4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff43c00a010 0x7ff43c00c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 msgr2=0x7ff43c009980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:13:26.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> v2:172.21.15.153:3300/0 conn(0x7ff43c0074f0 0x7ff43c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 reap_dead start 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c00dc40 con 0x7ff43c0038c0 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff462013700 5 monclient: authenticate success, global_id 4104 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7ff44400d780 con 0x7ff43c0038c0 2023-09-29T23:13:26.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff462013700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: dump: 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:26.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 >> v1:172.21.15.12:6789/0 conn(0x7ff43c0038c0 legacy=0x7ff43c005ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient: _add_conns ranks=[1,3,4,0,2,6,7,5,8] 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): picked mon.f con 0x7ff4480042b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448004e70 0x7ff4480052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): picked mon.g con 0x7ff448004e70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff448009c90 0x7ff44800c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): picked mon.c con 0x7ff448009c90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09e350 con 0x7ff4480042b0 2023-09-29T23:13:26.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09ef70 con 0x7ff448004e70 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff44800ca00 con 0x7ff448009c90 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45a7fc700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff462013700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff448009c90 0x7ff44800c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448004e70 0x7ff4480052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff4480042b0 auth_method 0 2023-09-29T23:13:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff448004e70 auth_method 0 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff448009c90 auth_method 0 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_done global_id 4104 payload 293 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff448009c90 msgr2=0x7ff44800c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff448009c90 0x7ff44800c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448004e70 msgr2=0x7ff4480052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448004e70 0x7ff4480052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 monclient: found mon.f 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff448003570 con 0x7ff4480042b0 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 msgr2=0x7ff448004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 msgr2=0x7ff448004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.372+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7ff4440134a0 tx=0x7ff444008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:13:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.373+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:13:26.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient: get_auth_request con 0x7ff4480042b0 auth_method 0 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 0 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 msgr2=0x7ff448004810 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480042b0 0x7ff448004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient: _add_conns ranks=[5,7,0,6,8,2] 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff44800d690 0x7ff44800fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): picked mon.h con 0x7ff44800d690 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:26.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff448010140 0x7ff448012520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): picked mon.i con 0x7ff448010140 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 0x7ff4480150d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): picked mon.a con 0x7ff448012c40 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448015a10 con 0x7ff448012c40 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff448015b50 con 0x7ff44800d690 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff448015d10 con 0x7ff448010140 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:26.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45a7fc700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 0x7ff4480150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff448012c40 auth_method 0 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff44800d690 0x7ff44800fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff448010140 0x7ff448012520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff44800d690 auth_method 0 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff448010140 auth_method 0 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.574+0000 7ff45bfff700 10 monclient(hunting): handle_auth_done global_id 4104 payload 293 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff448010140 msgr2=0x7ff448012520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff448010140 0x7ff448012520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff44800d690 msgr2=0x7ff44800fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7ff44800d690 0x7ff44800fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 monclient: found mon.a 2023-09-29T23:13:26.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7ff45c04c960 con 0x7ff448012c40 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 10 monclient: _finish_auth 0 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 -- 172.21.15.12:0/2985315052 reap_dead start 2023-09-29T23:13:26.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45bfff700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 0x7ff4480150d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff45c06aeb0 tx=0x7ff454009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fdf11377c70026 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff454001c40 con 0x7ff448012c40 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 10 monclient: dump: 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:26.589 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff462013700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 1 -- 172.21.15.12:0/2985315052 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45401c070 con 0x7ff448012c40 2023-09-29T23:13:26.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff45a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff462013700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.575+0000 7ff462013700 4 set_mon_vals no callback set 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 10 monclient: shutdown 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 -- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 msgr2=0x7ff4480150d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 0x7ff4480150d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff45c06aeb0 tx=0x7ff454009040 comp rx=0 tx=0).stop 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 -- 172.21.15.12:0/2985315052 shutdown_connections 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 --2- 172.21.15.12:0/2985315052 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff448012c40 0x7ff4480150d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 -- 172.21.15.12:0/2985315052 >> 172.21.15.12:0/2985315052 conn(0x7ff45c0a3b50 msgr2=0x7ff45c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 -- 172.21.15.12:0/2985315052 shutdown_connections 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 1 -- 172.21.15.12:0/2985315052 wait complete. 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 10 monclient: build_initial_monmap 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.576+0000 7ff462013700 10 monclient: monmap: 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:26.592 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:26.593 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 1 Processor -- start 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 1 -- start start 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 10 monclient: init 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 10 monclient: _add_conns ranks=[3,4,0,7,1,5,8,2,6] 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.577+0000 7ff462013700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448012c40 0x7ff45c132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): picked mon.noname-g con 0x7ff448012c40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): picked mon.noname-c con 0x7ff45c133010 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): picked mon.noname-a con 0x7ff45c13e370 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c12e7e0 con 0x7ff45c13e370 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09ffc0 con 0x7ff448012c40 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c09e510 con 0x7ff45c133010 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff462013700 10 monclient(hunting): authenticate will time out at 1115.938080s 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54030/0 (socket says 172.21.15.12:54030) 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 -- 172.21.15.12:0/3072845347 learned_addr learned my addr 172.21.15.12:0/3072845347 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): get_auth_request con 0x7ff45c13e370 auth_method 0 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:60750/0 (socket says 172.21.15.12:60750) 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): get_auth_request con 0x7ff45c133010 auth_method 0 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:26.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45b7fe700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448012c40 0x7ff45c132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request con 0x7ff448012c40 auth_method 0 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient(hunting): handle_auth_done global_id 4113 payload 824 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 -- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 msgr2=0x7ff45c13de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 -- 172.21.15.12:0/3072845347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448012c40 msgr2=0x7ff45c132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448012c40 0x7ff45c132ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:26.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 monclient: found mon.noname-a 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 -- 172.21.15.12:0/3072845347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c12fcf0 con 0x7ff45c13e370 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45affd700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff45bfff700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009fe0 tx=0x7ff45401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff454028070 con 0x7ff45c13e370 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff4537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff4537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:26.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff4537fe700 10 monclient: dump: 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:26.600 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.578+0000 7ff4537fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454019cb0 con 0x7ff45c13e370 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:26.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff454019e20 con 0x7ff45c13e370 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 10 monclient: dump: 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:26.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 5 monclient: authenticate success, global_id 4113 2023-09-29T23:13:26.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff459ffb700 4 set_mon_vals no callback set 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 10 monclient: _renew_subs 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c12ffd0 con 0x7ff45c13e370 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 10 monclient: _renew_subs 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c13e7e0 con 0x7ff45c13e370 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff454019830 con 0x7ff45c13e370 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.579+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff454021ad0 con 0x7ff45c13e370 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.580+0000 7ff462013700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:26.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.580+0000 7ff462013700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.580+0000 7ff462013700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.580+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c070bc0 con 0x7ff45c13e370 2023-09-29T23:13:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.582+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff45403a060 con 0x7ff45c13e370 2023-09-29T23:13:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.582+0000 7ff4537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:26.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.582+0000 7ff4537fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:26.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.659+0000 7ff462013700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-29T23:13:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.659+0000 7ff462013700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-29T23:13:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.659+0000 7ff462013700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:26.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.659+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff45c12f960 con 0x7ff45c13e370 2023-09-29T23:13:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.663+0000 7ff4537fe700 1 -- 172.21.15.12:0/3072845347 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff454021ce0 con 0x7ff45c13e370 2023-09-29T23:13:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.663+0000 7ff4537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-29T23:13:26.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.663+0000 7ff4537fe700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 10 monclient: shutdown 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 msgr2=0x7ff45c12c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009fe0 tx=0x7ff45401f000 comp rx=0 tx=0).stop 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 shutdown_connections 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff45c13e370 0x7ff45c12c620 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff448012c40 0x7ff45c132ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 --2- 172.21.15.12:0/3072845347 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff45c133010 0x7ff45c13de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:26.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 >> 172.21.15.12:0/3072845347 conn(0x7ff45c0a3b50 msgr2=0x7ff45c0a5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 shutdown_connections 2023-09-29T23:13:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:26.664+0000 7ff462013700 1 -- 172.21.15.12:0/3072845347 wait complete. 2023-09-29T23:13:26.666 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2023-09-29T23:13:26.676 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-29T23:13:26.677 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:26.677 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-29T23:13:26.702 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:26.702 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-29T23:13:26.766 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-09-29T23:13:26.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-29T23:13:26.831 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:26.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-29T23:13:26.862 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:26.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-29T23:13:26.930 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-09-29T23:13:26.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-29T23:13:26.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 41e52849-9960-40e6-97ad-dc6b6ba4c787 0 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.157+0000 7f0853a66700 10 monclient: get_monmap_and_config 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.157+0000 7f0853a66700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.157+0000 7f0853a66700 10 monclient: monmap: 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:27.158696+0000 2023-09-29T23:13:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:27.158696+0000 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.160 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:27.160 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:27.160 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.160 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 Processor -- start 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 -- start start 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient: init 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient: _add_conns ranks=[8,7,0,2,10,3,4,1,6,5,9] 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f084c0a1f90 0x7f084c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-e con 0x7f084c0a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.170 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-i con 0x7f084c0a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f084c133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c068ed0 con 0x7f084c133740 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f084c09e350 con 0x7f084c0a2360 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f084c09ef70 con 0x7f084c0a1f90 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.158+0000 7f0853a66700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0853a66700 10 monclient(hunting): authenticate will time out at 1102.699008s 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:37750/0 (socket says 172.21.15.153:37750) 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 -- 172.21.15.153:0/1938980780 learned_addr learned my addr 172.21.15.153:0/1938980780 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.171 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): get_auth_request con 0x7f084c0a2360 auth_method 0 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f084c0a1f90 0x7f084c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): get_auth_request con 0x7f084c0a1f90 auth_method 0 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.172 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 -- 172.21.15.153:0/1938980780 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f084c0a1f90 msgr2=0x7f084c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f084c0a1f90 0x7f084c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 -- 172.21.15.153:0/1938980780 >> v1:172.21.15.12:6789/0 conn(0x7f084c133740 legacy=0x7f084c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 monclient: found mon.noname-i 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 1 -- 172.21.15.153:0/1938980780 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682925921 0 0) 0x7f084c068ed0 con 0x7f084c133740 2023-09-29T23:13:27.173 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 -- 172.21.15.153:0/1938980780 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c04c960 con 0x7f084c0a2360 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0851001700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0840009a10 tx=0x7f0840009250 comp rx=0 tx=0).ready entity=mon.7 client_cookie=165e8c03d8faddc1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 1 -- 172.21.15.153:0/1938980780 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f084000bca0 con 0x7f084c0a2360 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 10 monclient: dump: 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.175 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0853a66700 5 monclient: authenticate success, global_id 4120 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0850800700 1 -- 172.21.15.153:0/1938980780 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084000be20 con 0x7f084c0a2360 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.159+0000 7f0853a66700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0850800700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0850800700 1 -- 172.21.15.153:0/1938980780 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f08400118a0 con 0x7f084c0a2360 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0850800700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 4 set_mon_vals no callback set 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0850800700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0850800700 10 monclient: dump: 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.177 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 10 monclient: shutdown 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 -- 172.21.15.153:0/1938980780 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 msgr2=0x7f084c0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0840009a10 tx=0x7f0840009250 comp rx=0 tx=0).stop 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 -- 172.21.15.153:0/1938980780 shutdown_connections 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f084c0a1f90 0x7f084c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.178 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 --2- 172.21.15.153:0/1938980780 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f084c0a2360 0x7f084c0a27b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 -- 172.21.15.153:0/1938980780 >> 172.21.15.153:0/1938980780 conn(0x7f084c0a3b50 msgr2=0x7f084c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 -- 172.21.15.153:0/1938980780 shutdown_connections 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.160+0000 7f0853a66700 1 -- 172.21.15.153:0/1938980780 wait complete. 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.161+0000 7f0853a66700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.161+0000 7f0853a66700 10 monclient: monmap: 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.161+0000 7f0853a66700 1 Processor -- start 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.161+0000 7f0853a66700 1 -- start start 2023-09-29T23:13:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.161+0000 7f0853a66700 10 monclient: init 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient: _add_conns ranks=[0,5,6,2,7,3,1,4,8] 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-a con 0x7f084c0a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 0x7f084c09c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-h con 0x7f084c0a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f084c133740 0x7f084c09cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): picked mon.noname-d con 0x7f084c133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c09ffc0 con 0x7f084c0a1f90 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f084c130fe0 con 0x7f084c0a2360 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f084c130d00 con 0x7f084c133740 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0853a66700 10 monclient(hunting): authenticate will time out at 1102.702251s 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 0x7f084c09c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.153:41960/0 (socket says 172.21.15.153:41960) 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 -- 172.21.15.153:0/957398937 learned_addr learned my addr 172.21.15.153:0/957398937 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): get_auth_request con 0x7f084c0a1f90 auth_method 0 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): get_auth_request con 0x7f084c0a2360 auth_method 0 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f084c133740 0x7f084c09cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): get_auth_request con 0x7f084c133740 auth_method 0 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0852003700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 -- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f084c133740 msgr2=0x7f084c09cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f084c133740 0x7f084c09cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 -- 172.21.15.153:0/957398937 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 msgr2=0x7f084c09c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 0x7f084c09c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 monclient: found mon.noname-a 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 -- 172.21.15.153:0/957398937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400094f0 con 0x7f084c0a1f90 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851001700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 0x7f084c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.162+0000 7f0851802700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0844009e50 tx=0x7f084400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0844014480 con 0x7f084c0a1f90 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: dump: 2023-09-29T23:13:27.185 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.186 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 5 monclient: authenticate success, global_id 4131 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400f040 con 0x7f084c0a1f90 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0844014ac0 con 0x7f084c0a1f90 2023-09-29T23:13:27.187 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 10 monclient: _renew_subs 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c12fac0 con 0x7f084c0a1f90 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f084bfff700 4 set_mon_vals no callback set 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 10 monclient: dump: 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.188 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0844003020 con 0x7f084c0a1f90 2023-09-29T23:13:27.189 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 10 monclient: _renew_subs 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c131d70 con 0x7f084c0a1f90 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.163+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f084400cc30 con 0x7f084c0a1f90 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.164+0000 7f0853a66700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.164+0000 7f0853a66700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.164+0000 7f0853a66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.164+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c070bc0 con 0x7f084c0a1f90 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.166+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0844018020 con 0x7f084c0a1f90 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.166+0000 7f0849ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.166+0000 7f0849ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.247+0000 7f0853a66700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "41e52849-9960-40e6-97ad-dc6b6ba4c787", "id": 0}] 2023-09-29T23:13:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.248+0000 7f0853a66700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "41e52849-9960-40e6-97ad-dc6b6ba4c787", "id": 0}] 2023-09-29T23:13:27.248 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.248+0000 7f0853a66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.249 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.248+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd new", "uuid": "41e52849-9960-40e6-97ad-dc6b6ba4c787", "id": 0} v 0) v1 -- 0x7f084c066790 con 0x7f084c0a1f90 2023-09-29T23:13:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.251+0000 7f0849ffb700 1 -- 172.21.15.153:0/957398937 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41e52849-9960-40e6-97ad-dc6b6ba4c787", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f084400cdf0 con 0x7f084c0a1f90 2023-09-29T23:13:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.251+0000 7f0849ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "41e52849-9960-40e6-97ad-dc6b6ba4c787", "id": 0}] 2023-09-29T23:13:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.251+0000 7f0849ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:27.252 INFO:teuthology.orchestra.run.smithi153.stdout:0 2023-09-29T23:13:27.253 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 10 monclient: shutdown 2023-09-29T23:13:27.253 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 msgr2=0x7f084c09c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0844009e50 tx=0x7f084400b040 comp rx=0 tx=0).stop 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 shutdown_connections 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f084c133740 0x7f084c09cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f084c0a1f90 0x7f084c09c480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 --2- 172.21.15.153:0/957398937 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f084c0a2360 0x7f084c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 >> 172.21.15.153:0/957398937 conn(0x7f084c0a3b50 msgr2=0x7f084c135e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 shutdown_connections 2023-09-29T23:13:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.253+0000 7f0853a66700 1 -- 172.21.15.153:0/957398937 wait complete. 2023-09-29T23:13:27.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 3765ea7f-51a2-489c-b4ba-c9aab2f2bd60 1 2023-09-29T23:13:27.384 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.383+0000 7f907c08a700 10 monclient: get_monmap_and_config 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.384+0000 7f907c08a700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.384+0000 7f907c08a700 10 monclient: monmap: 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:27.385204+0000 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:27.385204+0000 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.385 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.386 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:27.387 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:27.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.384+0000 7f907c08a700 1 Processor -- start 2023-09-29T23:13:27.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.384+0000 7f907c08a700 1 -- start start 2023-09-29T23:13:27.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.384+0000 7f907c08a700 10 monclient: init 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient: _add_conns ranks=[9,2,6,5,3,10,7,1,4,0,8] 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90740a1f90 0x7f907409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-a con 0x7f90740a1f90 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-b con 0x7f90740a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9074133740 0x7f9074133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-d con 0x7f9074133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907409e350 con 0x7f90740a1f90 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907409ef70 con 0x7f90740a2360 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f907409da70 con 0x7f9074133740 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907c08a700 10 monclient(hunting): authenticate will time out at 1102.925283s 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90740a1f90 0x7f907409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9074133740 0x7f9074133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.153:51964/0 (socket says 172.21.15.153:51964) 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 1 -- 172.21.15.153:0/3461778933 learned_addr learned my addr 172.21.15.153:0/3461778933 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): get_auth_request con 0x7f90740a2360 auth_method 0 2023-09-29T23:13:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f90740a1f90 0x7f907409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.153:42812/0 (socket says 172.21.15.153:42812) 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): get_auth_request con 0x7f90740a1f90 auth_method 0 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): get_auth_request con 0x7f9074133740 auth_method 0 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 -- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9074133740 msgr2=0x7f9074133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9074133740 0x7f9074133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 -- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 msgr2=0x7f90740a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 monclient: found mon.noname-a 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079e26700 1 -- 172.21.15.153:0/3461778933 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074068ed0 con 0x7f90740a1f90 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.385+0000 7f9079625700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9079e26700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90740a1f90 0x7f907409f5e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9068009910 tx=0x7f90680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa839ffaa4ecef0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 1 -- 172.21.15.153:0/3461778933 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9068011ab0 con 0x7f90740a1f90 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: dump: 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 5 monclient: authenticate success, global_id 4140 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 1 -- 172.21.15.153:0/3461778933 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068011c30 con 0x7f90740a1f90 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.402 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 1 -- 172.21.15.153:0/3461778933 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9068011da0 con 0x7f90740a1f90 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 4 set_mon_vals no callback set 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f9078e24700 10 monclient: dump: 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.403 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 10 monclient: shutdown 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 1 -- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90740a1f90 msgr2=0x7f907409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.386+0000 7f907c08a700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90740a1f90 0x7f907409f5e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9068009910 tx=0x7f90680091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 -- 172.21.15.153:0/3461778933 shutdown_connections 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9074133740 0x7f9074133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90740a1f90 0x7f907409f5e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 --2- 172.21.15.153:0/3461778933 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a2360 0x7f90740a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 -- 172.21.15.153:0/3461778933 >> 172.21.15.153:0/3461778933 conn(0x7f90740a3b50 msgr2=0x7f90740a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 -- 172.21.15.153:0/3461778933 shutdown_connections 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 -- 172.21.15.153:0/3461778933 wait complete. 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 10 monclient: monmap: 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:27.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.387+0000 7f907c08a700 1 Processor -- start 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 -- start start 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient: init 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient: _add_conns ranks=[2,4,1,8,7,5,0,6,3] 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a1f90 0x7f9074142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-b con 0x7f90740a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f90740a2360 0x7f90741329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-c con 0x7f90740a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): picked mon.noname-f con 0x7f9074133740 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907409ffc0 con 0x7f9074133740 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907412fd00 con 0x7f90740a1f90 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9074130bf0 con 0x7f90740a2360 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907c08a700 10 monclient(hunting): authenticate will time out at 1102.928477s 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36436/0 (socket says 172.21.15.153:36436) 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 -- 172.21.15.153:0/1606760352 learned_addr learned my addr 172.21.15.153:0/1606760352 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): get_auth_request con 0x7f9074133740 auth_method 0 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f90740a2360 0x7f90741329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a1f90 0x7f9074142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): get_auth_request con 0x7f90740a1f90 auth_method 0 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): get_auth_request con 0x7f90740a2360 auth_method 0 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.409 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079e26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f9079625700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 -- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f90740a2360 msgr2=0x7f90741329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f90740a2360 0x7f90741329c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.410 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 -- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a1f90 msgr2=0x7f9074142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a1f90 0x7f9074142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 monclient: found mon.noname-f 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.388+0000 7f907a627700 1 -- 172.21.15.153:0/1606760352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700094f0 con 0x7f9074133740 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907a627700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f906c009e90 tx=0x7f906c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f906c014480 con 0x7f9074133740 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: dump: 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.412 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 5 monclient: authenticate success, global_id 4132 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c00f040 con 0x7f9074133740 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 10 monclient: _renew_subs 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068009420 con 0x7f9074133740 2023-09-29T23:13:27.413 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f906c014aa0 con 0x7f9074133740 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 10 monclient: dump: 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 10 monclient: _renew_subs 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9067fff700 4 set_mon_vals no callback set 2023-09-29T23:13:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907404c960 con 0x7f9074133740 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f906c014c20 con 0x7f9074133740 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.389+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f906c01d7c0 con 0x7f9074133740 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.390+0000 7f907c08a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.390+0000 7f907c08a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.390+0000 7f907c08a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.390+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9074070bc0 con 0x7f9074133740 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.392+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f906c018020 con 0x7f9074133740 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.392+0000 7f9065ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.416 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.392+0000 7f9065ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.475+0000 7f907c08a700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3765ea7f-51a2-489c-b4ba-c9aab2f2bd60", "id": 1}] 2023-09-29T23:13:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.475+0000 7f907c08a700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3765ea7f-51a2-489c-b4ba-c9aab2f2bd60", "id": 1}] 2023-09-29T23:13:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.475+0000 7f907c08a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.475+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3765ea7f-51a2-489c-b4ba-c9aab2f2bd60", "id": 1} v 0) v1 -- 0x7f9074143050 con 0x7f9074133740 2023-09-29T23:13:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.481+0000 7f9065ffb700 1 -- 172.21.15.153:0/1606760352 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3765ea7f-51a2-489c-b4ba-c9aab2f2bd60", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f906c014670 con 0x7f9074133740 2023-09-29T23:13:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.481+0000 7f9065ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3765ea7f-51a2-489c-b4ba-c9aab2f2bd60", "id": 1}] 2023-09-29T23:13:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.481+0000 7f9065ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:27.482 INFO:teuthology.orchestra.run.smithi153.stdout:1 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 10 monclient: shutdown 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 msgr2=0x7f907412d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f906c009e90 tx=0x7f906c00b040 comp rx=0 tx=0).stop 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 shutdown_connections 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9074133740 0x7f907412d490 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90740a1f90 0x7f9074142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 --2- 172.21.15.153:0/1606760352 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f90740a2360 0x7f90741329c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 >> 172.21.15.153:0/1606760352 conn(0x7f90740a3b50 msgr2=0x7f90740a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 shutdown_connections 2023-09-29T23:13:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.482+0000 7f907c08a700 1 -- 172.21.15.153:0/1606760352 wait complete. 2023-09-29T23:13:27.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new c4d5f6b9-7c5d-40b9-9176-0930acb9cd20 2 2023-09-29T23:13:27.608 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.607+0000 7fc82ceb2700 10 monclient: get_monmap_and_config 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.607+0000 7fc82ceb2700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.608+0000 7fc82ceb2700 10 monclient: monmap: 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:27.609157+0000 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:27.609157+0000 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:27.609 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:27.610 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.608+0000 7fc82ceb2700 1 Processor -- start 2023-09-29T23:13:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.608+0000 7fc82ceb2700 1 -- start start 2023-09-29T23:13:27.611 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.608+0000 7fc82ceb2700 10 monclient: init 2023-09-29T23:13:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient: _add_conns ranks=[6,7,0,2,3,9,1,4,8,5,10] 2023-09-29T23:13:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc8280a1f90 0x7fc82809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-d con 0x7fc8280a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:27.619 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-i con 0x7fc8280a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc828133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc828068ed0 con 0x7fc828133740 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc82809e350 con 0x7fc8280a1f90 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc82809ef70 con 0x7fc8280a2360 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient(hunting): authenticate will time out at 1103.149196s 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.620 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:44660/0 (socket says 172.21.15.153:44660) 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 -- 172.21.15.153:0/315195902 learned_addr learned my addr 172.21.15.153:0/315195902 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): get_auth_request con 0x7fc8280a2360 auth_method 0 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82659c700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc8280a1f90 0x7fc82809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.621 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 -- 172.21.15.153:0/315195902 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc8280a1f90 msgr2=0x7fc82809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc8280a1f90 0x7fc82809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 -- 172.21.15.153:0/315195902 >> v1:172.21.15.12:6789/0 conn(0x7fc828133740 legacy=0x7fc828133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 monclient: found mon.noname-i 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 -- 172.21.15.153:0/315195902 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82804c960 con 0x7fc8280a2360 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc825d9b700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8180099e0 tx=0x7fc818009250 comp rx=0 tx=0).ready entity=mon.7 client_cookie=4ff8bdf2e71b99df server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 1 -- 172.21.15.153:0/315195902 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc81800bca0 con 0x7fc8280a2360 2023-09-29T23:13:27.622 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: dump: 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.623 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 1 -- 172.21.15.153:0/315195902 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81800be20 con 0x7fc8280a2360 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 5 monclient: authenticate success, global_id 4129 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 1 -- 172.21.15.153:0/315195902 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8180118a0 con 0x7fc8280a2360 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82ceb2700 4 set_mon_vals no callback set 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.609+0000 7fc82559a700 10 monclient: dump: 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 10 monclient: shutdown 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 -- 172.21.15.153:0/315195902 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 msgr2=0x7fc8280a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8180099e0 tx=0x7fc818009250 comp rx=0 tx=0).stop 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 -- 172.21.15.153:0/315195902 shutdown_connections 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc8280a1f90 0x7fc82809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/315195902 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc8280a2360 0x7fc8280a27b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 -- 172.21.15.153:0/315195902 >> 172.21.15.153:0/315195902 conn(0x7fc8280a3b50 msgr2=0x7fc8280a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 -- 172.21.15.153:0/315195902 shutdown_connections 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.610+0000 7fc82ceb2700 1 -- 172.21.15.153:0/315195902 wait complete. 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient: monmap: 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 Processor -- start 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 -- start start 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient: init 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient: _add_conns ranks=[1,4,7,8,0,5,2,6,3] 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-f con 0x7fc8280a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc8280a2360 0x7fc82809cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-c con 0x7fc8280a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): picked mon.noname-i con 0x7fc828133740 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828130ff0 con 0x7fc8280a1f90 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc828130a30 con 0x7fc8280a2360 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc828130d10 con 0x7fc828133740 2023-09-29T23:13:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82ceb2700 10 monclient(hunting): authenticate will time out at 1103.151879s 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc826d9d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.611+0000 7fc82659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:44672/0 (socket says 172.21.15.153:44672) 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36442/0 (socket says 172.21.15.153:36442) 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 -- 172.21.15.153:0/707936878 learned_addr learned my addr 172.21.15.153:0/707936878 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): get_auth_request con 0x7fc828133740 auth_method 0 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): get_auth_request con 0x7fc8280a1f90 auth_method 0 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc8280a2360 0x7fc82809cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): get_auth_request con 0x7fc8280a2360 auth_method 0 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc825d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 -- 172.21.15.153:0/707936878 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc8280a2360 msgr2=0x7fc82809cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc8280a2360 0x7fc82809cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 -- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 msgr2=0x7fc82809c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 monclient: found mon.noname-i 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 -- 172.21.15.153:0/707936878 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180094f0 con 0x7fc828133740 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82659c700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc826d9d700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc82000c960 tx=0x7fc8200026e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc820012ab0 con 0x7fc828133740 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: dump: 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 5 monclient: authenticate success, global_id 4138 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc820012c30 con 0x7fc828133740 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 10 monclient: _renew_subs 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82812fb40 con 0x7fc828133740 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc820012da0 con 0x7fc828133740 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc824d99700 4 set_mon_vals no callback set 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 10 monclient: dump: 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 10 monclient: _renew_subs 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82809d1e0 con 0x7fc828133740 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc82001b400 con 0x7fc828133740 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.612+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fc8200128b0 con 0x7fc828133740 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.613+0000 7fc82ceb2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.613+0000 7fc82ceb2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.613+0000 7fc82ceb2700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.613+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82809e490 con 0x7fc828133740 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.615+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc820016020 con 0x7fc828133740 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.615+0000 7fc81e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.615+0000 7fc81e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:27.698 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.698+0000 7fc82ceb2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c4d5f6b9-7c5d-40b9-9176-0930acb9cd20", "id": 2}] 2023-09-29T23:13:27.698 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.698+0000 7fc82ceb2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c4d5f6b9-7c5d-40b9-9176-0930acb9cd20", "id": 2}] 2023-09-29T23:13:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.698+0000 7fc82ceb2700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.699 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.698+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "osd new", "uuid": "c4d5f6b9-7c5d-40b9-9176-0930acb9cd20", "id": 2} v 0) v1 -- 0x7fc828070bc0 con 0x7fc828133740 2023-09-29T23:13:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.706+0000 7fc81e7fc700 1 -- 172.21.15.153:0/707936878 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4d5f6b9-7c5d-40b9-9176-0930acb9cd20", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc82001b6b0 con 0x7fc828133740 2023-09-29T23:13:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.706+0000 7fc81e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c4d5f6b9-7c5d-40b9-9176-0930acb9cd20", "id": 2}] 2023-09-29T23:13:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.706+0000 7fc81e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:27.707 INFO:teuthology.orchestra.run.smithi153.stdout:2 2023-09-29T23:13:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 10 monclient: shutdown 2023-09-29T23:13:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 msgr2=0x7fc82812d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc82000c960 tx=0x7fc8200026e0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 shutdown_connections 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.707+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8280a1f90 0x7fc82809c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.708+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc828133740 0x7fc82812d320 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.708+0000 7fc82ceb2700 1 --2- 172.21.15.153:0/707936878 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc8280a2360 0x7fc82809cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.708+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 >> 172.21.15.153:0/707936878 conn(0x7fc8280a3b50 msgr2=0x7fc8280a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.708+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 shutdown_connections 2023-09-29T23:13:27.709 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.708+0000 7fc82ceb2700 1 -- 172.21.15.153:0/707936878 wait complete. 2023-09-29T23:13:27.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92 3 2023-09-29T23:13:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.834+0000 7feb25896700 10 monclient: get_monmap_and_config 2023-09-29T23:13:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.834+0000 7feb25896700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.834+0000 7feb25896700 10 monclient: monmap: 2023-09-29T23:13:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.835 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:27.835830+0000 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:27.835830+0000 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 Processor -- start 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 -- start start 2023-09-29T23:13:27.837 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient: init 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient: _add_conns ranks=[9,4,6,0,8,1,3,10,2,5,7] 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): picked mon.noname-a con 0x7feb20130fe0 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): picked mon.noname-c con 0x7feb2006f0b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): picked mon.noname-d con 0x7feb200969e0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:27.847 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb201313b0 con 0x7feb20130fe0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7feb20131e00 con 0x7feb2006f0b0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feb2006d690 con 0x7feb200969e0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb25896700 10 monclient(hunting): authenticate will time out at 1103.375737s 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb1e7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb1effd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.835+0000 7feb1f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.153:46136/0 (socket says 172.21.15.153:46136) 2023-09-29T23:13:27.848 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.153:42820/0 (socket says 172.21.15.153:42820) 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 1 -- 172.21.15.153:0/2598482197 learned_addr learned my addr 172.21.15.153:0/2598482197 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.153:42924/0 (socket says 172.21.15.153:42924) 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): get_auth_request con 0x7feb200969e0 auth_method 0 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): get_auth_request con 0x7feb2006f0b0 auth_method 0 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.849 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): get_auth_request con 0x7feb20130fe0 auth_method 0 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.850 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 -- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 msgr2=0x7feb2006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 -- 172.21.15.153:0/2598482197 >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 msgr2=0x7feb20131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 --2- 172.21.15.153:0/2598482197 >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 monclient: found mon.noname-d 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 -- 172.21.15.153:0/2598482197 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb20068ed0 con 0x7feb200969e0 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1effd700 1 --2- 172.21.15.153:0/2598482197 >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1e7fc700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1f7fe700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb18009e60 tx=0x7feb18009780 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f716475852f6b8ab server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 1 -- 172.21.15.153:0/2598482197 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb1800e070 con 0x7feb200969e0 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: dump: 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.853 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 1 -- 172.21.15.153:0/2598482197 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18012430 con 0x7feb200969e0 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb25896700 5 monclient: authenticate success, global_id 4128 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb25896700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 1 -- 172.21.15.153:0/2598482197 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb18012a20 con 0x7feb200969e0 2023-09-29T23:13:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb25896700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb25896700 4 set_mon_vals no callback set 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.836+0000 7feb1dffb700 10 monclient: dump: 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.855 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.856 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 10 monclient: shutdown 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 -- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 msgr2=0x7feb20096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb18009e60 tx=0x7feb18009780 comp rx=0 tx=0).stop 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 -- 172.21.15.153:0/2598482197 shutdown_connections 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb200969e0 0x7feb20096e20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 --2- 172.21.15.153:0/2598482197 >> v2:172.21.15.12:3300/0 conn(0x7feb20130fe0 0x7feb20131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 --2- 172.21.15.153:0/2598482197 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feb2006f0b0 0x7feb2006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 -- 172.21.15.153:0/2598482197 >> 172.21.15.153:0/2598482197 conn(0x7feb200a4430 msgr2=0x7feb200a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.857 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 -- 172.21.15.153:0/2598482197 shutdown_connections 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.837+0000 7feb25896700 1 -- 172.21.15.153:0/2598482197 wait complete. 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.838+0000 7feb25896700 10 monclient: build_initial_monmap 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.838+0000 7feb25896700 10 monclient: monmap: 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:27.858 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:27.859 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.839+0000 7feb25896700 1 Processor -- start 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.839+0000 7feb25896700 1 -- start start 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.839+0000 7feb25896700 10 monclient: init 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.839+0000 7feb25896700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.839+0000 7feb25896700 10 monclient: _add_conns ranks=[7,0,6,4,8,3,1,2,5] 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): picked mon.noname-i con 0x7feb2006f0b0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 0x7feb2011aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): picked mon.noname-a con 0x7feb200969e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 0x7feb2011afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): picked mon.noname-d con 0x7feb20130fe0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2011e370 con 0x7feb200969e0 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feb20140f30 con 0x7feb20130fe0 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feb200a0180 con 0x7feb2006f0b0 2023-09-29T23:13:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient(hunting): authenticate will time out at 1103.380099s 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:44686/0 (socket says 172.21.15.153:44686) 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 -- 172.21.15.153:0/218297625 learned_addr learned my addr 172.21.15.153:0/218297625 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): get_auth_request con 0x7feb2006f0b0 auth_method 0 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 0x7feb2011afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 0x7feb2011aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 10 monclient(hunting): get_auth_request con 0x7feb20130fe0 auth_method 0 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 10 monclient(hunting): get_auth_request con 0x7feb200969e0 auth_method 0 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 -- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 msgr2=0x7feb2011afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 0x7feb2011afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 -- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 msgr2=0x7feb2011aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 0x7feb2011aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 monclient: found mon.noname-i 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 -- 172.21.15.153:0/218297625 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100094f0 con 0x7feb2006f0b0 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1e7fc700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 0x7feb2011aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:27.864 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1f7fe700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 0x7feb2011afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1effd700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb1400ecb0 tx=0x7feb140091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb14009780 con 0x7feb2006f0b0 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: dump: 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:27.866 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 5 monclient: authenticate success, global_id 4147 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1400cbe0 con 0x7feb2006f0b0 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb14013d70 con 0x7feb2006f0b0 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient: _renew_subs 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb14009420 con 0x7feb2006f0b0 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb1d7fa700 4 set_mon_vals no callback set 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb06ffd700 10 monclient: dump: 2023-09-29T23:13:27.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient: _renew_subs 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.840+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb18009a20 con 0x7feb2006f0b0 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7feb14005a00 con 0x7feb2006f0b0 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7feb14013b40 con 0x7feb2006f0b0 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb25896700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb25896700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb25896700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.869 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.841+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2004c960 con 0x7feb2006f0b0 2023-09-29T23:13:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.843+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7feb1403a6d0 con 0x7feb2006f0b0 2023-09-29T23:13:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.843+0000 7feb06ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:27.870 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.843+0000 7feb06ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:27.927 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.927+0000 7feb25896700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92", "id": 3}] 2023-09-29T23:13:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.927+0000 7feb25896700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92", "id": 3}] 2023-09-29T23:13:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.927+0000 7feb25896700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:27.928 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.927+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "osd new", "uuid": "da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92", "id": 3} v 0) v1 -- 0x7feb20070bc0 con 0x7feb2006f0b0 2023-09-29T23:13:27.938 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.935+0000 7feb06ffd700 1 -- 172.21.15.153:0/218297625 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb14029070 con 0x7feb2006f0b0 2023-09-29T23:13:27.938 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.935+0000 7feb06ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92", "id": 3}] 2023-09-29T23:13:27.938 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.935+0000 7feb06ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:27.938 INFO:teuthology.orchestra.run.smithi153.stdout:3 2023-09-29T23:13:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 10 monclient: shutdown 2023-09-29T23:13:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 msgr2=0x7feb2012df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb1400ecb0 tx=0x7feb140091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 shutdown_connections 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feb20130fe0 0x7feb2011afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb2006f0b0 0x7feb2012df40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 --2- 172.21.15.153:0/218297625 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb200969e0 0x7feb2011aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 >> 172.21.15.153:0/218297625 conn(0x7feb200a4430 msgr2=0x7feb200a2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 shutdown_connections 2023-09-29T23:13:27.940 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:27.936+0000 7feb25896700 1 -- 172.21.15.153:0/218297625 wait complete. 2023-09-29T23:13:27.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 558fc6a6-cfca-4177-9514-ff36bb7d87bc 4 2023-09-29T23:13:28.064 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 10 monclient: get_monmap_and_config 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 10 monclient: monmap: 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:28.065344+0000 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:28.065344+0000 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:28.065 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:28.066 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:28.067 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.067 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 1 Processor -- start 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 1 -- start start 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.064+0000 7fde178b8700 10 monclient: init 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient: _add_conns ranks=[5,9,7,4,2,3,10,0,6,8,1] 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-h con 0x7fde1006ece0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 0x7fde1006f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-a con 0x7fde10133c80 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:28.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-i con 0x7fde1006f5f0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1009deb0 con 0x7fde10133c80 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fde1009f290 con 0x7fde1006ece0 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fde1009f570 con 0x7fde1006f5f0 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde178b8700 10 monclient(hunting): authenticate will time out at 1103.605394s 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.153:43636/0 (socket says 172.21.15.153:43636) 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:44702/0 (socket says 172.21.15.153:44702) 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 -- 172.21.15.153:0/4073412058 learned_addr learned my addr 172.21.15.153:0/4073412058 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): get_auth_request con 0x7fde1006ece0 auth_method 0 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): get_auth_request con 0x7fde1006f5f0 auth_method 0 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 1 --2- 172.21.15.153:0/4073412058 >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 0x7fde1006f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 10 monclient(hunting): get_auth_request con 0x7fde10133c80 auth_method 0 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.079 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 -- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 msgr2=0x7fde100698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 -- 172.21.15.153:0/4073412058 >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 msgr2=0x7fde1006f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 --2- 172.21.15.153:0/4073412058 >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 0x7fde1006f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 monclient: found mon.noname-h 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15654700 1 -- 172.21.15.153:0/4073412058 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10068ed0 con 0x7fde1006ece0 2023-09-29T23:13:28.080 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde14e53700 1 --2- 172.21.15.153:0/4073412058 >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 0x7fde1006f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.065+0000 7fde15e55700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde15654700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde040099b0 tx=0x7fde04009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=b4b825f2f8a917ce server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 1 -- 172.21.15.153:0/4073412058 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde04009ea0 con 0x7fde1006ece0 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: dump: 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.081 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.082 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 5 monclient: authenticate success, global_id 4118 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 1 -- 172.21.15.153:0/4073412058 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400bce0 con 0x7fde1006ece0 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 4 set_mon_vals no callback set 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 1 -- 172.21.15.153:0/4073412058 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde0400be50 con 0x7fde1006ece0 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde03fff700 10 monclient: dump: 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.084 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 10 monclient: shutdown 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 -- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 msgr2=0x7fde10133740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde040099b0 tx=0x7fde04009250 comp rx=0 tx=0).stop 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 -- 172.21.15.153:0/4073412058 shutdown_connections 2023-09-29T23:13:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fde1006f5f0 0x7fde100698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 --2- 172.21.15.153:0/4073412058 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fde1006ece0 0x7fde10133740 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 --2- 172.21.15.153:0/4073412058 >> v2:172.21.15.12:3300/0 conn(0x7fde10133c80 0x7fde1006f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 -- 172.21.15.153:0/4073412058 >> 172.21.15.153:0/4073412058 conn(0x7fde1006aab0 msgr2=0x7fde1006aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 -- 172.21.15.153:0/4073412058 shutdown_connections 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.066+0000 7fde178b8700 1 -- 172.21.15.153:0/4073412058 wait complete. 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.067+0000 7fde178b8700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.067+0000 7fde178b8700 10 monclient: monmap: 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.067+0000 7fde178b8700 1 Processor -- start 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.067+0000 7fde178b8700 1 -- start start 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.067+0000 7fde178b8700 10 monclient: init 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient: _add_conns ranks=[0,8,1,5,3,2,6,7,4] 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 0x7fde10142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-a con 0x7fde1006ece0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 0x7fde1012ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-e con 0x7fde1006f5f0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): picked mon.noname-f con 0x7fde10133c80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:28.088 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1009ebe0 con 0x7fde1006ece0 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1009e810 con 0x7fde10133c80 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fde10120350 con 0x7fde1006f5f0 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde178b8700 10 monclient(hunting): authenticate will time out at 1103.608344s 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36450/0 (socket says 172.21.15.153:36450) 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 -- 172.21.15.153:0/1064726879 learned_addr learned my addr 172.21.15.153:0/1064726879 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:28.089 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): get_auth_request con 0x7fde10133c80 auth_method 0 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde14e53700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 0x7fde1012ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 -- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 msgr2=0x7fde1012ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.090 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 0x7fde1012ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15654700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 0x7fde10142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 -- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 msgr2=0x7fde10142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 0x7fde10142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 monclient: found mon.noname-f 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 -- 172.21.15.153:0/1064726879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08009a20 con 0x7fde10133c80 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde14e53700 10 monclient: get_auth_request con 0x7fde1006f5f0 auth_method 0 2023-09-29T23:13:28.091 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15654700 10 monclient: get_auth_request con 0x7fde1006ece0 auth_method 0 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde14e53700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 0x7fde1012ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15654700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 0x7fde10142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.068+0000 7fde15e55700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c0f0 tx=0x7fde0800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde08009d20 con 0x7fde10133c80 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: dump: 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.092 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.093 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 5 monclient: authenticate success, global_id 4150 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0800f040 con 0x7fde10133c80 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde08014d80 con 0x7fde10133c80 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 10 monclient: _renew_subs 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde037fe700 4 set_mon_vals no callback set 2023-09-29T23:13:28.094 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde040094f0 con 0x7fde10133c80 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 10 monclient: dump: 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.095 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.096 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 10 monclient: _renew_subs 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1004c960 con 0x7fde10133c80 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fde08003020 con 0x7fde10133c80 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.069+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fde08014980 con 0x7fde10133c80 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.070+0000 7fde178b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.070+0000 7fde178b8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.070+0000 7fde178b8700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.070+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde10070bc0 con 0x7fde10133c80 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.072+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fde08018030 con 0x7fde10133c80 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.072+0000 7fde017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.097 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.072+0000 7fde017fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:28.154 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.154+0000 7fde178b8700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "558fc6a6-cfca-4177-9514-ff36bb7d87bc", "id": 4}] 2023-09-29T23:13:28.154 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.154+0000 7fde178b8700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "558fc6a6-cfca-4177-9514-ff36bb7d87bc", "id": 4}] 2023-09-29T23:13:28.155 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.154+0000 7fde178b8700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:28.155 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.154+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "558fc6a6-cfca-4177-9514-ff36bb7d87bc", "id": 4} v 0) v1 -- 0x7fde1009f850 con 0x7fde10133c80 2023-09-29T23:13:28.164 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.164+0000 7fde017fa700 1 -- 172.21.15.153:0/1064726879 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "558fc6a6-cfca-4177-9514-ff36bb7d87bc", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fde0800ce80 con 0x7fde10133c80 2023-09-29T23:13:28.165 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.164+0000 7fde017fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "558fc6a6-cfca-4177-9514-ff36bb7d87bc", "id": 4}] 2023-09-29T23:13:28.165 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.164+0000 7fde017fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:28.165 INFO:teuthology.orchestra.run.smithi153.stdout:4 2023-09-29T23:13:28.166 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 10 monclient: shutdown 2023-09-29T23:13:28.166 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.166 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 msgr2=0x7fde1011dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.166 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c0f0 tx=0x7fde0800b040 comp rx=0 tx=0).stop 2023-09-29T23:13:28.166 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 shutdown_connections 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde10133c80 0x7fde1011dda0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fde1006ece0 0x7fde10142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 --2- 172.21.15.153:0/1064726879 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fde1006f5f0 0x7fde1012ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 >> 172.21.15.153:0/1064726879 conn(0x7fde1006aab0 msgr2=0x7fde100a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 shutdown_connections 2023-09-29T23:13:28.167 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.165+0000 7fde178b8700 1 -- 172.21.15.153:0/1064726879 wait complete. 2023-09-29T23:13:28.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new c0b953bb-ccc4-4fc2-a85a-15f012096923 5 2023-09-29T23:13:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.291+0000 7f7d3f7b0700 10 monclient: get_monmap_and_config 2023-09-29T23:13:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.291+0000 7f7d3f7b0700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.291+0000 7f7d3f7b0700 10 monclient: monmap: 2023-09-29T23:13:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:28.293124+0000 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:28.293124+0000 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.293 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.294 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.294 INFO:teuthology.orchestra.run.smithi153.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:28.294 INFO:teuthology.orchestra.run.smithi153.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:28.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.292+0000 7f7d3f7b0700 1 Processor -- start 2023-09-29T23:13:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.292+0000 7f7d3f7b0700 1 -- start start 2023-09-29T23:13:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.292+0000 7f7d3f7b0700 10 monclient: init 2023-09-29T23:13:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient: _add_conns ranks=[4,6,8,1,0,3,7,5,9,10,2] 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-c con 0x7f7d3806ba40 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 0x7f7d3806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-d con 0x7f7d3806e8b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-e con 0x7f7d380969e0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3806d8f0 con 0x7f7d3806ba40 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.304 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3806c890 con 0x7f7d3806e8b0 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3809ec80 con 0x7f7d380969e0 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3f7b0700 10 monclient(hunting): authenticate will time out at 1103.833286s 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.153:44274/0 (socket says 172.21.15.153:44274) 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.153:46144/0 (socket says 172.21.15.153:46144) 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 1 -- 172.21.15.153:0/4156940114 learned_addr learned my addr 172.21.15.153:0/4156940114 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 0x7f7d3806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.305 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): get_auth_request con 0x7f7d380969e0 auth_method 0 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): get_auth_request con 0x7f7d3806ba40 auth_method 0 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): get_auth_request con 0x7f7d3806e8b0 auth_method 0 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_done global_id 4162 payload 274 2023-09-29T23:13:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 msgr2=0x7f7d381363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 msgr2=0x7f7d3806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 0x7f7d3806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 monclient: found mon.noname-c 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4156940114 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38068ed0 con 0x7f7d3806ba40 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3dd4d700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.293+0000 7f7d3cd4b700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 0x7f7d3806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:28.308 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009910 tx=0x7f7d280091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6ee131bdc98e54e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 1 -- 172.21.15.153:0/4156940114 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d28011ab0 con 0x7f7d3806ba40 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: dump: 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.310 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d3f7b0700 5 monclient: authenticate success, global_id 4162 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 1 -- 172.21.15.153:0/4156940114 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28011c30 con 0x7f7d3806ba40 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 1 -- 172.21.15.153:0/4156940114 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d28011da0 con 0x7f7d3806ba40 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d2ffff700 10 monclient: dump: 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.311 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.312 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d3f7b0700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.294+0000 7f7d3f7b0700 4 set_mon_vals no callback set 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 10 monclient: shutdown 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 msgr2=0x7f7d3806be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009910 tx=0x7f7d280091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4156940114 shutdown_connections 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d381363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7d3806ba40 0x7f7d3806be10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4156940114 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7d3806e8b0 0x7f7d3806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4156940114 >> 172.21.15.153:0/4156940114 conn(0x7f7d3806a370 msgr2=0x7f7d380a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4156940114 shutdown_connections 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4156940114 wait complete. 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 10 monclient: monmap: 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.295+0000 7f7d3f7b0700 1 Processor -- start 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 -- start start 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient: init 2023-09-29T23:13:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient: _add_conns ranks=[7,3,8,0,1,2,4,6,5] 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-i con 0x7f7d3806ba40 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-g con 0x7f7d3806e8b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d38098c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): picked mon.noname-e con 0x7f7d380969e0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3806ecb0 con 0x7f7d3806e8b0 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3813ecb0 con 0x7f7d3806ba40 2023-09-29T23:13:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3809c660 con 0x7f7d380969e0 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3f7b0700 10 monclient(hunting): authenticate will time out at 1103.836718s 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44754/0 (socket says 172.21.15.153:44754) 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.153:44704/0 (socket says 172.21.15.153:44704) 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 1 -- 172.21.15.153:0/4069891473 learned_addr learned my addr 172.21.15.153:0/4069891473 (peer_addr_for_me v2:172.21.15.153:0/0) 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): get_auth_request con 0x7f7d3806e8b0 auth_method 0 2023-09-29T23:13:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): get_auth_request con 0x7f7d3806ba40 auth_method 0 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3dd4d700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d38098c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3dd4d700 10 monclient(hunting): get_auth_request con 0x7f7d380969e0 auth_method 0 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3dd4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3dd4d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3dd4d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.296+0000 7f7d3cd4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3dd4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2023-09-29T23:13:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4069891473 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 msgr2=0x7f7d38098c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d38098c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 msgr2=0x7f7d3812b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 monclient: found mon.noname-i 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 -- 172.21.15.153:0/4069891473 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34009420 con 0x7f7d3806ba40 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3cd4b700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3d54c700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d280098e0 tx=0x7f7d28017000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d2801f070 con 0x7f7d3806ba40 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: dump: 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28002f30 con 0x7f7d3806ba40 2023-09-29T23:13:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 5 monclient: authenticate success, global_id 4165 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 10 monclient: _renew_subs 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3000bcf0 con 0x7f7d3806ba40 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7d28006a90 con 0x7f7d3806ba40 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2f7fe700 4 set_mon_vals no callback set 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 10 monclient: dump: 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 10 monclient: _renew_subs 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d28009420 con 0x7f7d3806ba40 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7d28023070 con 0x7f7d3806ba40 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.297+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f7d2802fa20 con 0x7f7d3806ba40 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.298+0000 7f7d3f7b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.298+0000 7f7d3f7b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.298+0000 7f7d3f7b0700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.298+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3804c960 con 0x7f7d3806ba40 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.300+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7d2802a070 con 0x7f7d3806ba40 2023-09-29T23:13:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.300+0000 7f7d2d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.326 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.300+0000 7f7d2d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:28.365 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:28.365+0000 7fe8a15dc180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T23:13:28.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.386+0000 7f7d3f7b0700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c0b953bb-ccc4-4fc2-a85a-15f012096923", "id": 5}] 2023-09-29T23:13:28.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.386+0000 7f7d3f7b0700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c0b953bb-ccc4-4fc2-a85a-15f012096923", "id": 5}] 2023-09-29T23:13:28.387 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.386+0000 7f7d3f7b0700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:28.388 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.386+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "osd new", "uuid": "c0b953bb-ccc4-4fc2-a85a-15f012096923", "id": 5} v 0) v1 -- 0x7f7d3809ca60 con 0x7f7d3806ba40 2023-09-29T23:13:28.393 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.393+0000 7f7d2d7fa700 1 -- 172.21.15.153:0/4069891473 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c0b953bb-ccc4-4fc2-a85a-15f012096923", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7d28015020 con 0x7f7d3806ba40 2023-09-29T23:13:28.393 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.393+0000 7f7d2d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c0b953bb-ccc4-4fc2-a85a-15f012096923", "id": 5}] 2023-09-29T23:13:28.393 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.393+0000 7f7d2d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:28.393 INFO:teuthology.orchestra.run.smithi153.stdout:5 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 10 monclient: shutdown 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 msgr2=0x7f7d3813de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d280098e0 tx=0x7f7d28017000 comp rx=0 tx=0).stop 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 shutdown_connections 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7d380969e0 0x7f7d38098c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7d3806ba40 0x7f7d3813de10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 --2- 172.21.15.153:0/4069891473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d3806e8b0 0x7f7d3812b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 >> 172.21.15.153:0/4069891473 conn(0x7f7d3806a370 msgr2=0x7f7d381319e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.395 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 shutdown_connections 2023-09-29T23:13:28.396 INFO:teuthology.orchestra.run.smithi153.stderr:2023-09-29T23:13:28.394+0000 7f7d3f7b0700 1 -- 172.21.15.153:0/4069891473 wait complete. 2023-09-29T23:13:28.406 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-29T23:13:28.406 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-29T23:13:28.408 INFO:tasks.ceph.osd.0:Started 2023-09-29T23:13:28.409 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-29T23:13:28.409 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-29T23:13:28.411 INFO:tasks.ceph.osd.1:Started 2023-09-29T23:13:28.411 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-29T23:13:28.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-29T23:13:28.413 INFO:tasks.ceph.osd.2:Started 2023-09-29T23:13:28.413 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-29T23:13:28.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-29T23:13:28.415 INFO:tasks.ceph.osd.3:Started 2023-09-29T23:13:28.415 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-29T23:13:28.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-29T23:13:28.417 INFO:tasks.ceph.osd.4:Started 2023-09-29T23:13:28.417 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-29T23:13:28.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-29T23:13:28.419 INFO:tasks.ceph.osd.5:Started 2023-09-29T23:13:28.420 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:13:28.489 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:28.488+0000 7f4f1c17e540 -1 Falling back to public interface 2023-09-29T23:13:28.490 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.489+0000 7f9ec71d8540 -1 Falling back to public interface 2023-09-29T23:13:28.490 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.489+0000 7f9ec1600700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6802/0: (98) Address already in use 2023-09-29T23:13:28.490 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:28.490+0000 7f0f71370540 -1 Falling back to public interface 2023-09-29T23:13:28.498 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:28.497+0000 7ff590d23540 -1 Falling back to public interface 2023-09-29T23:13:28.498 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.498+0000 7ff15fb9e540 -1 Falling back to public interface 2023-09-29T23:13:28.498 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.498+0000 7ff159fc6700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-09-29T23:13:28.498 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:28.498+0000 7f0fc9cd3540 -1 Falling back to public interface 2023-09-29T23:13:28.500 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f9ec71d8540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.501 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f4f1c17e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.501 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f9ec71d8540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.501 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f9ec71d8540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.502 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f4f1c17e540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.502 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:28.500+0000 7f4f1c17e540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.503 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:28.502+0000 7f0f71370540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.503 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:28.502+0000 7f0f71370540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.503 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:28.502+0000 7f0f71370540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.507 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:28.507+0000 7f4f1c17e540 -1 osd.3 0 log_to_monitors true 2023-09-29T23:13:28.508 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:28.508+0000 7f9ec71d8540 -1 osd.4 0 log_to_monitors true 2023-09-29T23:13:28.510 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:28.510+0000 7f0f71370540 -1 osd.5 0 log_to_monitors true 2023-09-29T23:13:28.510 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:28.510+0000 7ff590d23540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.510 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:28.510+0000 7f0fc9cd3540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.511 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:28.510+0000 7ff590d23540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.511 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:28.510+0000 7ff590d23540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.511 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:28.510+0000 7f0fc9cd3540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.511 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:28.510+0000 7f0fc9cd3540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.514 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.514+0000 7ff15fb9e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-29T23:13:28.514 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.514+0000 7ff15fb9e540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.515 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.514+0000 7ff15fb9e540 -1 journal do_read_entry(8192): bad header magic 2023-09-29T23:13:28.517 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:28.517+0000 7fe8a15dc180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T23:13:28.517 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:28.517+0000 7ff590d23540 -1 osd.1 0 log_to_monitors true 2023-09-29T23:13:28.517 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:28.517+0000 7f0fc9cd3540 -1 osd.2 0 log_to_monitors true 2023-09-29T23:13:28.522 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:28.522+0000 7ff15fb9e540 -1 osd.0 0 log_to_monitors true 2023-09-29T23:13:28.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.562+0000 7f1f129e5700 10 monclient: get_monmap_and_config 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.562+0000 7f1f129e5700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.562+0000 7f1f129e5700 10 monclient: monmap: 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:28.563471+0000 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:28.563471+0000 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:28.563 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.564 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:28.565 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:28.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 Processor -- start 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 -- start start 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient: init 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient: _add_conns ranks=[6,10,0,7,3,8,1,2,5,9,4] 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-d con 0x7f1f0c09b410 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1f0c09a9a0 0x7f1f0c0a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-f con 0x7f1f0c09a9a0 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f0c09b7e0 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:28.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0c068ed0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0c09c720 con 0x7f1f0c09a9a0 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0a26d0 con 0x7f1f0c09b410 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f129e5700 10 monclient(hunting): authenticate will time out at 1117.923886s 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.563+0000 7f1f0bfff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f10f82700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1f0c09b7e0 0x7f1f0c0a1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36800/0 (socket says 172.21.15.12:36800) 2023-09-29T23:13:28.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f10f82700 1 -- 172.21.15.12:0/3804069337 learned_addr learned my addr 172.21.15.12:0/3804069337 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:51498/0 (socket says 172.21.15.12:51498) 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): get_auth_request con 0x7f1f0c09b410 auth_method 0 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 1 --2- 172.21.15.12:0/3804069337 >> v2:172.21.15.153:3300/0 conn(0x7f1f0c09a9a0 0x7f1f0c0a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): get_auth_request con 0x7f1f0c09a9a0 auth_method 0 2023-09-29T23:13:28.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 -- 172.21.15.12:0/3804069337 >> v2:172.21.15.153:3300/0 conn(0x7f1f0c09a9a0 msgr2=0x7f1f0c0a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 --2- 172.21.15.12:0/3804069337 >> v2:172.21.15.153:3300/0 conn(0x7f1f0c09a9a0 0x7f1f0c0a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 1 -- 172.21.15.12:0/3804069337 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272976709 0 0) 0x7f1f0c068ed0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 -- 172.21.15.12:0/3804069337 >> v1:172.21.15.12:6789/0 conn(0x7f1f0c09b7e0 legacy=0x7f1f0c0a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 monclient: found mon.noname-d 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 -- 172.21.15.12:0/3804069337 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c04c960 con 0x7f1f0c09b410 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0bfff700 1 --2- 172.21.15.12:0/3804069337 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009910 tx=0x7f1f000091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=60034883ef0feb3d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 1 -- 172.21.15.12:0/3804069337 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1f0000bac0 con 0x7f1f0c09b410 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: dump: 2023-09-29T23:13:28.577 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.578 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f129e5700 5 monclient: authenticate success, global_id 4191 2023-09-29T23:13:28.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 1 -- 172.21.15.12:0/3804069337 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0000bc40 con 0x7f1f0c09b410 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f129e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f129e5700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f129e5700 4 set_mon_vals no callback set 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 1 -- 172.21.15.12:0/3804069337 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1f0000bdb0 con 0x7f1f0c09b410 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.564+0000 7f1f0affd700 10 monclient: dump: 2023-09-29T23:13:28.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 10 monclient: shutdown 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 -- 172.21.15.12:0/3804069337 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 msgr2=0x7f1f0c09cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 --2- 172.21.15.12:0/3804069337 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009910 tx=0x7f1f000091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 -- 172.21.15.12:0/3804069337 shutdown_connections 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 --2- 172.21.15.12:0/3804069337 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1f0c09b410 0x7f1f0c09cb20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 --2- 172.21.15.12:0/3804069337 >> v2:172.21.15.153:3300/0 conn(0x7f1f0c09a9a0 0x7f1f0c0a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 -- 172.21.15.12:0/3804069337 >> 172.21.15.12:0/3804069337 conn(0x7f1f0c063d70 msgr2=0x7f1f0c09fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 -- 172.21.15.12:0/3804069337 shutdown_connections 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 1 -- 172.21.15.12:0/3804069337 wait complete. 2023-09-29T23:13:28.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 10 monclient: build_initial_monmap 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.565+0000 7f1f129e5700 10 monclient: monmap: 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 Processor -- start 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 -- start start 2023-09-29T23:13:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient: init 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient: _add_conns ranks=[5,4,0,1,6,2,8,7,3] 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1f0c09a9a0 0x7f1f0c132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-h con 0x7f1f0c09a9a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 0x7f1f0c11c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-c con 0x7f1f0c09b410 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): picked mon.noname-a con 0x7f1f0c09b7e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c11fbf0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c099a10 con 0x7f1f0c09b410 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c11e5d0 con 0x7f1f0c09a9a0 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f129e5700 10 monclient(hunting): authenticate will time out at 1117.926802s 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0b7fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 0x7f1f0c11c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54064/0 (socket says 172.21.15.12:54064) 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 1 -- 172.21.15.12:0/29442325 learned_addr learned my addr 172.21.15.12:0/29442325 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 10 monclient(hunting): get_auth_request con 0x7f1f0c09b7e0 auth_method 0 2023-09-29T23:13:28.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f10f82700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0b7fe700 10 monclient(hunting): get_auth_request con 0x7f1f0c09b410 auth_method 0 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.566+0000 7f1f0bfff700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1f0c09a9a0 0x7f1f0c132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): get_auth_request con 0x7f1f0c09a9a0 auth_method 0 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:28.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 -- 172.21.15.12:0/29442325 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1f0c09a9a0 msgr2=0x7f1f0c132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1f0c09a9a0 0x7f1f0c132ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 -- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 msgr2=0x7f1f0c11c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 0x7f1f0c11c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient: _finish_hunting 0 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 monclient: found mon.noname-a 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 -- 172.21.15.12:0/29442325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00009420 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0b7fe700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 0x7f1f0c11c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f10f82700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400cc40 tx=0x7f1f04001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:28.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1f0400ac90 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: dump: 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 5 monclient: authenticate success, global_id 4212 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0400ae30 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 10 monclient: _renew_subs 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1f04007570 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0c06f580 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f0a7fc700 4 set_mon_vals no callback set 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 10 monclient: dump: 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:28.593 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:28.594 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 10 monclient: _renew_subs 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0c11ec90 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1f040070f0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.567+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f1f040073a0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.568+0000 7f1f129e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.568+0000 7f1f129e5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.568+0000 7f1f129e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.568+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0c070bc0 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.570+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1f0400a810 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.570+0000 7f1ef3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:28.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.570+0000 7f1ef3fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:28.596 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:28.594+0000 7fe8a15dc180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T23:13:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1f129e5700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1f129e5700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1f129e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:28.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f0c11e050 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1ef3fff700 1 -- 172.21.15.12:0/29442325 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f1f04007790 con 0x7f1f0c09b7e0 2023-09-29T23:13:28.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1ef3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:28.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.640+0000 7f1ef3fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:28.641 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:28.641 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":8,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:28.389101+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:13:28.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 10 monclient: shutdown 2023-09-29T23:13:28.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:28.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 msgr2=0x7f1f0c11caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:28.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400cc40 tx=0x7f1f04001970 comp rx=0 tx=0).stop 2023-09-29T23:13:28.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 shutdown_connections 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f0c09b7e0 0x7f1f0c11caa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1f0c09a9a0 0x7f1f0c132ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 --2- 172.21.15.12:0/29442325 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1f0c09b410 0x7f1f0c11c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 >> 172.21.15.12:0/29442325 conn(0x7f1f0c063d70 msgr2=0x7f1f0c09db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 shutdown_connections 2023-09-29T23:13:28.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:28.641+0000 7f1f129e5700 1 -- 172.21.15.12:0/29442325 wait complete. 2023-09-29T23:13:28.653 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-29T23:13:28.653 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T23:13:28.779 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:28.779+0000 7fe8a15dc180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:13:28.953 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:28.953+0000 7fe8a15dc180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T23:13:29.030 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.029+0000 7fe8a15dc180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T23:13:29.166 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.164+0000 7fe8a15dc180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T23:13:29.257 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:13:29.259 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.259+0000 7fe8a15dc180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T23:13:29.370 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.370+0000 7f775c165700 10 monclient: get_monmap_and_config 2023-09-29T23:13:29.370 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.370+0000 7f775c165700 10 monclient: build_initial_monmap 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.370+0000 7f775c165700 10 monclient: monmap: 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:29.371355+0000 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:29.371355+0000 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:29.371 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.372 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 Processor -- start 2023-09-29T23:13:29.374 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 -- start start 2023-09-29T23:13:29.374 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient: init 2023-09-29T23:13:29.374 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:29.374 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient: _add_conns ranks=[6,1,9,5,8,3,7,2,4,10,0] 2023-09-29T23:13:29.374 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): picked mon.noname-d con 0x7f77540a1460 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f775409fd10 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f77540a1f40 0x7f775413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): picked mon.noname-a con 0x7f77540a1f40 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7754066f20 con 0x7f775409fd10 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775409dc80 con 0x7f77540a1f40 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f77540a3070 con 0x7f77540a1460 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:29.375 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.371+0000 7f775c165700 10 monclient(hunting): authenticate will time out at 1118.731960s 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:51506/0 (socket says 172.21.15.12:51506) 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 -- 172.21.15.12:0/2515064273 learned_addr learned my addr 172.21.15.12:0/2515064273 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): get_auth_request con 0x7f77540a1460 auth_method 0 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:29.376 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 1 --2- 172.21.15.12:0/2515064273 >> v2:172.21.15.12:3300/0 conn(0x7f77540a1f40 0x7f775413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): get_auth_request con 0x7f77540a1f40 auth_method 0 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 -- 172.21.15.12:0/2515064273 >> v2:172.21.15.12:3300/0 conn(0x7f77540a1f40 msgr2=0x7f775413ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 --2- 172.21.15.12:0/2515064273 >> v2:172.21.15.12:3300/0 conn(0x7f77540a1f40 0x7f775413ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 -- 172.21.15.12:0/2515064273 >> v1:172.21.15.153:6789/0 conn(0x7f775409fd10 legacy=0x7f77540a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient: _finish_hunting 0 2023-09-29T23:13:29.377 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 monclient: found mon.noname-d 2023-09-29T23:13:29.378 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:29.378 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:29.378 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 -- 172.21.15.12:0/2515064273 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775404c960 con 0x7f77540a1460 2023-09-29T23:13:29.378 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.372+0000 7f7759f01700 1 --2- 172.21.15.12:0/2515064273 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7750009990 tx=0x7f7750009220 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4bca390663a3e22 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:29.383 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 1 -- 172.21.15.12:0/2515064273 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7750009e80 con 0x7f77540a1460 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: dump: 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:29.384 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: _finish_auth 0 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f775c165700 5 monclient: authenticate success, global_id 4200 2023-09-29T23:13:29.385 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f775c165700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 1 -- 172.21.15.12:0/2515064273 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750005c10 con 0x7f77540a1460 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f775c165700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f775c165700 4 set_mon_vals no callback set 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 1 -- 172.21.15.12:0/2515064273 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7750005d80 con 0x7f77540a1460 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.376+0000 7f7758eff700 10 monclient: dump: 2023-09-29T23:13:29.386 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:29.387 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 10 monclient: shutdown 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 -- 172.21.15.12:0/2515064273 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 msgr2=0x7f775409f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 --2- 172.21.15.12:0/2515064273 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7750009990 tx=0x7f7750009220 comp rx=0 tx=0).stop 2023-09-29T23:13:29.388 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 -- 172.21.15.12:0/2515064273 shutdown_connections 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 --2- 172.21.15.12:0/2515064273 >> v2:172.21.15.12:3300/0 conn(0x7f77540a1f40 0x7f775413ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 --2- 172.21.15.12:0/2515064273 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77540a1460 0x7f775409f7d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 -- 172.21.15.12:0/2515064273 >> 172.21.15.12:0/2515064273 conn(0x7f77540a67a0 msgr2=0x7f77540a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 -- 172.21.15.12:0/2515064273 shutdown_connections 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 1 -- 172.21.15.12:0/2515064273 wait complete. 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 10 monclient: build_initial_monmap 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.377+0000 7f775c165700 10 monclient: monmap: 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:29.389 INFO:teuthology.misc.health.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:created 0.000000 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:29.390 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 Processor -- start 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 -- start start 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient: init 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient: _add_conns ranks=[0,8,5,4,6,3,2,1,7] 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): picked mon.noname-a con 0x7f775409fd10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77540a1460 0x7f775412fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): picked mon.noname-e con 0x7f77540a1460 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:29.391 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77540a1f40 0x7f775406bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): picked mon.noname-h con 0x7f77540a1f40 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775406de60 con 0x7f775409fd10 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f775406e330 con 0x7f77540a1f40 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f775406d200 con 0x7f77540a1460 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775c165700 10 monclient(hunting): authenticate will time out at 1118.738738s 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77540a1460 0x7f775412fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:29.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54082/0 (socket says 172.21.15.12:54082) 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 1 -- 172.21.15.12:0/3822126137 learned_addr learned my addr 172.21.15.12:0/3822126137 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): get_auth_request con 0x7f775409fd10 auth_method 0 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): get_auth_request con 0x7f77540a1460 auth_method 0 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:29.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775a702700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77540a1f40 0x7f775406bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759f01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775a702700 10 monclient(hunting): get_auth_request con 0x7f77540a1f40 auth_method 0 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775a702700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775a702700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f775a702700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.378+0000 7f7759700700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:29.394 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775a702700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 -- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77540a1460 msgr2=0x7f775412fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77540a1460 0x7f775412fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 -- 172.21.15.12:0/3822126137 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77540a1f40 msgr2=0x7f775406bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77540a1f40 0x7f775406bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 10 monclient: _finish_hunting 0 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 monclient: found mon.noname-a 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:29.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 -- 172.21.15.12:0/3822126137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750009480 con 0x7f775409fd10 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7759f01700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7750009260 tx=0x7f7750016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f775001c440 con 0x7f775409fd10 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: dump: 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.396 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:29.397 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775001c5e0 con 0x7f775409fd10 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 5 monclient: authenticate success, global_id 4230 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 10 monclient: _renew_subs 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7754068f00 con 0x7f775409fd10 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:29.398 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f775001cbd0 con 0x7f775409fd10 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 10 monclient: dump: 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:29.399 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7747fff700 4 set_mon_vals no callback set 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 10 monclient: _renew_subs 2023-09-29T23:13:29.400 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775406f160 con 0x7f775409fd10 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f775001cd30 con 0x7f775409fd10 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.379+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f7750011ae0 con 0x7f775409fd10 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.380+0000 7f775c165700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.380+0000 7f775c165700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.380+0000 7f775c165700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.380+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77540a5fa0 con 0x7f775409fd10 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.382+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7750006c10 con 0x7f775409fd10 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.382+0000 7f7745ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:29.401 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.382+0000 7f7745ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.453+0000 7f775c165700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.453+0000 7f775c165700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.453+0000 7f775c165700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.453+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7754064420 con 0x7f775409fd10 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f7745ffb700 1 -- 172.21.15.12:0/3822126137 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f7750011d40 con 0x7f775409fd10 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f7745ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:29.454 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f7745ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:29.455 INFO:teuthology.misc.health.smithi012.stdout: 2023-09-29T23:13:29.455 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":9,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:29.390254+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 10 monclient: shutdown 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 msgr2=0x7f7754142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7750009260 tx=0x7f7750016000 comp rx=0 tx=0).stop 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 shutdown_connections 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77540a1f40 0x7f775406bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f775409fd10 0x7f7754142690 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.456 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 --2- 172.21.15.12:0/3822126137 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77540a1460 0x7f775412fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:29.457 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.454+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 >> 172.21.15.12:0/3822126137 conn(0x7f77540a67a0 msgr2=0x7f7754132a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:29.457 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.455+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 shutdown_connections 2023-09-29T23:13:29.457 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:29.455+0000 7f775c165700 1 -- 172.21.15.12:0/3822126137 wait complete. 2023-09-29T23:13:29.465 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-09-29T23:13:29.786 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.786+0000 7fe8a15dc180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T23:13:29.863 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.863+0000 7fe8a15dc180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T23:13:29.940 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:29.939+0000 7fe8a15dc180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T23:13:30.398 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:30.398+0000 7ff156fc0700 -1 osd.0 0 waiting for initial osdmap 2023-09-29T23:13:30.398 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:30.398+0000 7ff588145700 -1 osd.1 0 waiting for initial osdmap 2023-09-29T23:13:30.402 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-09-29T23:13:30.401+0000 7ff1535be700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.404 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:30.404+0000 7f4f135a0700 -1 osd.3 0 waiting for initial osdmap 2023-09-29T23:13:30.405 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:30.404+0000 7f9ebe5fa700 -1 osd.4 0 waiting for initial osdmap 2023-09-29T23:13:30.405 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:30.404+0000 7f0fc10f5700 -1 osd.2 0 waiting for initial osdmap 2023-09-29T23:13:30.405 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:30.404+0000 7f0f68792700 -1 osd.5 0 waiting for initial osdmap 2023-09-29T23:13:30.406 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-09-29T23:13:30.406+0000 7f0fbd6f3700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.406 INFO:tasks.ceph.osd.5.smithi153.stderr:2023-09-29T23:13:30.406+0000 7f0f64d90700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.407 INFO:tasks.ceph.osd.3.smithi153.stderr:2023-09-29T23:13:30.407+0000 7f4f0fb9e700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.408 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-09-29T23:13:30.407+0000 7f9ebabf8700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.413 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-09-29T23:13:30.413+0000 7ff584743700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:13:30.539 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:30.538+0000 7fe8a15dc180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T23:13:30.676 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:30.675+0000 7fe8a15dc180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:13:30.760 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:30.760+0000 7fe8a15dc180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T23:13:31.009 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:31.008+0000 7fe8a15dc180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T23:13:31.087 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:31.086+0000 7fe8a15dc180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T23:13:31.921 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:31.921+0000 7fe8a15dc180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T23:13:31.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:31.995+0000 7fe8a15dc180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T23:13:32.070 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.069+0000 7fe8a15dc180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T23:13:32.202 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.202+0000 7fe8a15dc180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T23:13:32.324 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.324+0000 7fe8a15dc180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T23:13:32.478 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.478+0000 7fe8a15dc180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T23:13:32.553 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.553+0000 7fe8a15dc180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T23:13:32.627 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.627+0000 7fe8a15dc180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T23:13:32.702 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.702+0000 7fe8a15dc180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T23:13:32.748 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.747+0000 7fe8318d1700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:13:32.748 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.747+0000 7fe8318d1700 -1 pg_autoscaler.serve: 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:32.747+0000 7fe8318d1700 -1 Traceback (most recent call last): 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr: self._maybe_adjust() 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr: if self.noautoscale: 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:13:32.749 INFO:tasks.ceph.mgr.x.smithi153.stderr: 2023-09-29T23:13:33.732 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.731+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.i 2023-09-29T23:13:33.732 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.732+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.d 2023-09-29T23:13:33.732 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.732+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.g 2023-09-29T23:13:33.733 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.733+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.h 2023-09-29T23:13:33.734 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.733+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.f 2023-09-29T23:13:33.734 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.734+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.e 2023-09-29T23:13:33.735 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.734+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-29T23:13:33.735 INFO:tasks.ceph.mgr.x.smithi153.stderr:2023-09-29T23:13:33.734+0000 7fe85a9e2700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-29T23:13:33.735 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:13:33.735+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-29T23:13:36.069 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.181+0000 7f012f3ec700 10 monclient: get_monmap_and_config 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.181+0000 7f012f3ec700 10 monclient: build_initial_monmap 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.181+0000 7f012f3ec700 10 monclient: monmap: 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:36.182544+0000 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:36.182544+0000 2023-09-29T23:13:36.182 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:36.183 INFO:teuthology.misc.health.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:36.184 INFO:teuthology.misc.health.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:36.184 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.184 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 Processor -- start 2023-09-29T23:13:36.184 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 -- start start 2023-09-29T23:13:36.184 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient: init 2023-09-29T23:13:36.192 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:36.192 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient: _add_conns ranks=[4,6,7,0,10,1,5,9,3,2,8] 2023-09-29T23:13:36.192 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-c con 0x7f01280a0e80 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-d con 0x7f01280a1250 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0128133780 0x7f0128133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-i con 0x7f0128133780 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f01280a2360 con 0x7f01280a0e80 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f012806aca0 con 0x7f01280a1250 2023-09-29T23:13:36.193 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f012809ede0 con 0x7f0128133780 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.182+0000 7f012f3ec700 10 monclient(hunting): authenticate will time out at 1125.542924s 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56174/0 (socket says 172.21.15.12:56174) 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37774/0 (socket says 172.21.15.12:37774) 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 1 -- 172.21.15.12:0/3798825361 learned_addr learned my addr 172.21.15.12:0/3798825361 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): get_auth_request con 0x7f01280a1250 auth_method 0 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.194 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): get_auth_request con 0x7f01280a0e80 auth_method 0 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0128133780 0x7f0128133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.195 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): get_auth_request con 0x7f0128133780 auth_method 0 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 -- 172.21.15.12:0/3798825361 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0128133780 msgr2=0x7f0128133bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:36.196 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0128133780 0x7f0128133bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 -- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 msgr2=0x7f01280a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient: _finish_hunting 0 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 monclient: found mon.noname-c 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 -- 172.21.15.12:0/3798825361 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128066f20 con 0x7f01280a0e80 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012dbe9700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012e3ea700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0124009900 tx=0x7f0124009120 comp rx=0 tx=0).ready entity=mon.4 client_cookie=63f2f9dfba1b47df server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.197 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 1 -- 172.21.15.12:0/3798825361 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0124005c10 con 0x7f01280a0e80 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: dump: 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.198 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: _finish_auth 0 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:36.199 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012f3ec700 5 monclient: authenticate success, global_id 4270 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012f3ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 1 -- 172.21.15.12:0/3798825361 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124005d90 con 0x7f01280a0e80 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012f3ec700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 1 -- 172.21.15.12:0/3798825361 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0124011420 con 0x7f01280a0e80 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012f3ec700 4 set_mon_vals no callback set 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.183+0000 7f012d3e8700 10 monclient: dump: 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.200 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.201 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 10 monclient: shutdown 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 -- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 msgr2=0x7f012809f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0124009900 tx=0x7f0124009120 comp rx=0 tx=0).stop 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 -- 172.21.15.12:0/3798825361 shutdown_connections 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0128133780 0x7f0128133bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01280a0e80 0x7f012809f450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 --2- 172.21.15.12:0/3798825361 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a1250 0x7f01280a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.202 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 -- 172.21.15.12:0/3798825361 >> 172.21.15.12:0/3798825361 conn(0x7f01280a3080 msgr2=0x7f01280a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 -- 172.21.15.12:0/3798825361 shutdown_connections 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.184+0000 7f012f3ec700 1 -- 172.21.15.12:0/3798825361 wait complete. 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient: build_initial_monmap 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient: monmap: 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:epoch 0 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:created 0.000000 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:36.203 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 1 Processor -- start 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 1 -- start start 2023-09-29T23:13:36.204 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient: init 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient: _add_conns ranks=[6,2,3,4,7,5,1,0,8] 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-d con 0x7f01280a0e80 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01280a1250 0x7f0128132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-b con 0x7f01280a1250 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.185+0000 7f012f3ec700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0128133780 0x7f012812b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): picked mon.noname-g con 0x7f0128133780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012809e380 con 0x7f01280a1250 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.205 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012809fe30 con 0x7f0128133780 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f012812de50 con 0x7f01280a0e80 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient(hunting): authenticate will time out at 1125.546030s 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01280a1250 0x7f0128132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56188/0 (socket says 172.21.15.12:56188) 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 -- 172.21.15.12:0/562395450 learned_addr learned my addr 172.21.15.12:0/562395450 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): get_auth_request con 0x7f01280a0e80 auth_method 0 2023-09-29T23:13:36.206 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): get_auth_request con 0x7f01280a1250 auth_method 0 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0128133780 0x7f012812b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): get_auth_request con 0x7f0128133780 auth_method 0 2023-09-29T23:13:36.207 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012dbe9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012ebeb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2023-09-29T23:13:36.208 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 -- 172.21.15.12:0/562395450 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0128133780 msgr2=0x7f012812b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0128133780 0x7f012812b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 -- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01280a1250 msgr2=0x7f0128132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01280a1250 0x7f0128132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient: _finish_hunting 0 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 monclient: found mon.noname-d 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 -- 172.21.15.12:0/562395450 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01180094f0 con 0x7f01280a0e80 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012e3ea700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0124009ef0 tx=0x7f0124016000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0124006ad0 con 0x7f01280a0e80 2023-09-29T23:13:36.209 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: dump: 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.210 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 5 monclient: authenticate success, global_id 4245 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012401aab0 con 0x7f01280a0e80 2023-09-29T23:13:36.211 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient: _renew_subs 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f012401ac60 con 0x7f01280a0e80 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01240093c0 con 0x7f01280a0e80 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012cbe7700 4 set_mon_vals no callback set 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f011e7fc700 10 monclient: dump: 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:epoch 1 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.212 INFO:teuthology.misc.health.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.213 INFO:teuthology.misc.health.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr: 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient: _renew_subs 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.186+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012804c960 con 0x7f01280a0e80 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 4) v1 ==== 42450+0+0 (secure 0 0 0) 0x7f012401a630 con 0x7f01280a0e80 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f012f3ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f012f3ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f012f3ec700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128068f00 con 0x7f01280a0e80 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f011e7fc700 1 --2- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 0x7f010c038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.214 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.187+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f0124048410 con 0x7f01280a0e80 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.188+0000 7f012dbe9700 1 --2- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 0x7f010c038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.188+0000 7f012dbe9700 10 monclient: get_auth_request con 0x7f010c0367c0 auth_method 0 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.188+0000 7f012dbe9700 1 --2- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 0x7f010c038c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0118006d40 tx=0x7f0118009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.192+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f012401a8e0 con 0x7f01280a0e80 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.192+0000 7f011e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.215 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.192+0000 7f011e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:36.392 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.391+0000 7f012f3ec700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f012f3ec700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f012f3ec700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01280a4d40 con 0x7f01280a0e80 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f011e7fc700 1 -- 172.21.15.12:0/562395450 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8457 (secure 0 0 0) 0x7f012402eb40 con 0x7f01280a0e80 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f011e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.392+0000 7f011e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:36.393 INFO:teuthology.misc.health.smithi012.stdout: 2023-09-29T23:13:36.394 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":15,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:35.744677+0000","last_up_change":"2023-09-29T23:13:31.402198+0000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:13:32.778797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":83761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":83761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":83761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":83761}]},"public_addr":"172.21.15.12:6802/83761","cluster_addr":"172.21.15.12:6804/83761","heartbeat_back_addr":"172.21.15.12:6807/83761","heartbeat_front_addr":"172.21.15.12:6806/83761","state":["exists","up"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":83762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6801","nonce":83762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6805","nonce":83762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":83762}]},"public_addr":"172.21.15.12:6800/83762","cluster_addr":"172.21.15.12:6801/83762","heartbeat_back_addr":"172.21.15.12:6805/83762","heartbeat_front_addr":"172.21.15.12:6803/83762","state":["exists","up"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":83763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":83763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":83763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":83763}]},"public_addr":"172.21.15.12:6808/83763","cluster_addr":"172.21.15.12:6809/83763","heartbeat_back_addr":"172.21.15.12:6811/83763","heartbeat_front_addr":"172.21.15.12:6810/83763","state":["exists","up"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":83793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":83793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":83793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":83793}]},"public_addr":"172.21.15.153:6800/83793","cluster_addr":"172.21.15.153:6801/83793","heartbeat_back_addr":"172.21.15.153:6804/83793","heartbeat_front_addr":"172.21.15.153:6802/83793","state":["exists","up"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":83794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":83794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":83794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":83794}]},"public_addr":"172.21.15.153:6803/83794","cluster_addr":"172.21.15.153:6805/83794","heartbeat_back_addr":"172.21.15.153:6807/83794","heartbeat_front_addr":"172.21.15.153:6806/83794","state":["exists","up"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":83795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":83795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":83795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":83795}]},"public_addr":"172.21.15.153:6808/83795","cluster_addr":"172.21.15.153:6809/83795","heartbeat_back_addr":"172.21.15.153:6811/83795","heartbeat_front_addr":"172.21.15.153:6810/83795","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.551171+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.484006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.491551+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.520272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.501043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.509993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 msgr2=0x7f010c038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 0x7f010c038c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0118006d40 tx=0x7f0118009250 comp rx=0 tx=0).stop 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 10 monclient: shutdown 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 msgr2=0x7f012806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.395 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0124009ef0 tx=0x7f0124016000 comp rx=0 tx=0).stop 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 shutdown_connections 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01280a0e80 0x7f012806eaa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> v2:172.21.15.153:6812/83257 conn(0x7f010c0367c0 0x7f010c038c70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0128133780 0x7f012812b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 --2- 172.21.15.12:0/562395450 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f01280a1250 0x7f0128132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 >> 172.21.15.12:0/562395450 conn(0x7f01280a3080 msgr2=0x7f0128136080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 shutdown_connections 2023-09-29T23:13:36.396 INFO:teuthology.misc.health.smithi012.stderr:2023-09-29T23:13:36.394+0000 7f012f3ec700 1 -- 172.21.15.12:0/562395450 wait complete. 2023-09-29T23:13:36.405 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-09-29T23:13:36.405 INFO:tasks.ceph:Creating RBD pool 2023-09-29T23:13:36.405 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.521+0000 7fc79eacd700 10 monclient: get_monmap_and_config 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.521+0000 7fc79eacd700 10 monclient: build_initial_monmap 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.522+0000 7fc79eacd700 10 monclient: monmap: 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.522 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:36.522848+0000 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:36.522848+0000 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:36.523 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:36.524 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:36.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.522+0000 7fc79eacd700 1 Processor -- start 2023-09-29T23:13:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.522+0000 7fc79eacd700 1 -- start start 2023-09-29T23:13:36.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.522+0000 7fc79eacd700 10 monclient: init 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient: _add_conns ranks=[4,2,3,0,5,1,10,9,6,7,8] 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-c con 0x7fc7980a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-b con 0x7fc7980a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc798133700 0x7fc798133b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-g con 0x7fc798133700 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:36.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79809ed90 con 0x7fc7980a2360 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79809eab0 con 0x7fc798133700 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc79809df40 con 0x7fc7980a1f90 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79eacd700 10 monclient(hunting): authenticate will time out at 1125.883264s 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37782/0 (socket says 172.21.15.12:37782) 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34676/0 (socket says 172.21.15.12:34676) 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 -- 172.21.15.12:0/2806623132 learned_addr learned my addr 172.21.15.12:0/2806623132 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): get_auth_request con 0x7fc7980a2360 auth_method 0 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): get_auth_request con 0x7fc7980a1f90 auth_method 0 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc798133700 0x7fc798133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): get_auth_request con 0x7fc798133700 auth_method 0 2023-09-29T23:13:36.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 -- 172.21.15.12:0/2806623132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc798133700 msgr2=0x7fc798133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc798133700 0x7fc798133b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 -- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 msgr2=0x7fc7980a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient: _finish_hunting 0 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 monclient: found mon.noname-c 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 -- 172.21.15.12:0/2806623132 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798068ed0 con 0x7fc7980a1f90 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc797fff700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.523+0000 7fc79c869700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0099c0 tx=0x7fc78c009220 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ef196fdb219b2db2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 1 -- 172.21.15.12:0/2806623132 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc78c005c10 con 0x7fc7980a1f90 2023-09-29T23:13:36.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: dump: 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.540 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:36.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 5 monclient: authenticate success, global_id 4279 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 1 -- 172.21.15.12:0/2806623132 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c005db0 con 0x7fc7980a1f90 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 4 set_mon_vals no callback set 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 1 -- 172.21.15.12:0/2806623132 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc78c00b470 con 0x7fc7980a1f90 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:36.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc7977fe700 10 monclient: dump: 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 10 monclient: shutdown 2023-09-29T23:13:36.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 -- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 msgr2=0x7fc79809f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0099c0 tx=0x7fc78c009220 comp rx=0 tx=0).stop 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 -- 172.21.15.12:0/2806623132 shutdown_connections 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc7980a1f90 0x7fc79809f490 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc798133700 0x7fc798133b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 --2- 172.21.15.12:0/2806623132 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc7980a2360 0x7fc7980a2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 -- 172.21.15.12:0/2806623132 >> 172.21.15.12:0/2806623132 conn(0x7fc7980a6ff0 msgr2=0x7fc7980a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 -- 172.21.15.12:0/2806623132 shutdown_connections 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.524+0000 7fc79eacd700 1 -- 172.21.15.12:0/2806623132 wait complete. 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.525+0000 7fc79eacd700 10 monclient: build_initial_monmap 2023-09-29T23:13:36.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.525+0000 7fc79eacd700 10 monclient: monmap: 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:36.546 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.525+0000 7fc79eacd700 1 Processor -- start 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.525+0000 7fc79eacd700 1 -- start start 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.525+0000 7fc79eacd700 10 monclient: init 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient: _add_conns ranks=[5,6,1,3,7,8,2,0,4] 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc7980a1f90 0x7fc798142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-h con 0x7fc7980a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-d con 0x7fc7980a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc798133700 0x7fc79812fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): picked mon.noname-f con 0x7fc798133700 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79809e190 con 0x7fc798133700 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc79808c520 con 0x7fc7980a1f90 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc79808d4d0 con 0x7fc7980a2360 2023-09-29T23:13:36.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient(hunting): authenticate will time out at 1125.886176s 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56198/0 (socket says 172.21.15.12:56198) 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 -- 172.21.15.12:0/3064878961 learned_addr learned my addr 172.21.15.12:0/3064878961 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): get_auth_request con 0x7fc7980a2360 auth_method 0 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc798133700 0x7fc79812fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc7980a1f90 0x7fc798142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): get_auth_request con 0x7fc798133700 auth_method 0 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): get_auth_request con 0x7fc7980a1f90 auth_method 0 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:36.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79d06a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79c869700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 -- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc7980a1f90 msgr2=0x7fc798142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc7980a1f90 0x7fc798142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 -- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc798133700 msgr2=0x7fc79812fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:36.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc798133700 0x7fc79812fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 monclient: found mon.noname-d 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 -- 172.21.15.12:0/3064878961 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc784009420 con 0x7fc7980a2360 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc797fff700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc78400ec80 tx=0x7fc7840091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc78400cda0 con 0x7fc7980a2360 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: dump: 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:36.552 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.553 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: _finish_auth 0 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784013470 con 0x7fc7980a2360 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc784013a60 con 0x7fc7980a2360 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 5 monclient: authenticate success, global_id 4254 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:36.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc796ffd700 4 set_mon_vals no callback set 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc794ff9700 10 monclient: dump: 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:36.555 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient: _renew_subs 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.526+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c009480 con 0x7fc7980a2360 2023-09-29T23:13:36.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 10 monclient: _renew_subs 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79804c960 con 0x7fc7980a2360 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 4) v1 ==== 42450+0+0 (secure 0 0 0) 0x7fc784013bc0 con 0x7fc7980a2360 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.527+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79808c9c0 con 0x7fc7980a2360 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.528+0000 7fc794ff9700 1 --2- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 0x7fc77c038ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.528+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7fc784047870 con 0x7fc7980a2360 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.528+0000 7fc79c869700 1 --2- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 0x7fc77c038ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:36.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.528+0000 7fc79c869700 10 monclient: get_auth_request con 0x7fc77c0366f0 auth_method 0 2023-09-29T23:13:36.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.528+0000 7fc79c869700 1 --2- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 0x7fc77c038ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc78c000c00 tx=0x7fc78c016000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:36.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.532+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7840138c0 con 0x7fc7980a2360 2023-09-29T23:13:36.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.532+0000 7fc794ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:36.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.532+0000 7fc794ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.725+0000 7fc79eacd700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-29T23:13:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.725+0000 7fc79eacd700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-29T23:13:36.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.725+0000 7fc79eacd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:36.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:36.725+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fc798132660 con 0x7fc7980a2360 2023-09-29T23:13:37.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.120+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc78401c450 con 0x7fc7980a2360 2023-09-29T23:13:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.121+0000 7fc794ff9700 1 -- 172.21.15.12:0/3064878961 <== mon.6 v2:172.21.15.12:3303/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7fc78401a360 con 0x7fc7980a2360 2023-09-29T23:13:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.121+0000 7fc794ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-29T23:13:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.121+0000 7fc794ff9700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-29T23:13:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 msgr2=0x7fc77c038ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 0x7fc77c038ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc78c000c00 tx=0x7fc78c016000 comp rx=0 tx=0).stop 2023-09-29T23:13:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 10 monclient: shutdown 2023-09-29T23:13:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:37.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 msgr2=0x7fc798143030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.125+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc78400ec80 tx=0x7fc7840091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 shutdown_connections 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> v2:172.21.15.153:6812/83257 conn(0x7fc77c0366f0 0x7fc77c038ba0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc798133700 0x7fc79812fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc7980a1f90 0x7fc798142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 --2- 172.21.15.12:0/3064878961 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc7980a2360 0x7fc798143030 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 >> 172.21.15.12:0/3064878961 conn(0x7fc7980a6ff0 msgr2=0x7fc79806e300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 shutdown_connections 2023-09-29T23:13:37.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.126+0000 7fc79eacd700 1 -- 172.21.15.12:0/3064878961 wait complete. 2023-09-29T23:13:37.128 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2023-09-29T23:13:37.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T23:13:37.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.258+0000 7f527a528700 10 monclient: get_monmap_and_config 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.258+0000 7f527a528700 10 monclient: build_initial_monmap 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.258+0000 7f527a528700 10 monclient: monmap: 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:37.259514+0000 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:37.259514+0000 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:37.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:37.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 Processor -- start 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 -- start start 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient: init 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient: _add_conns ranks=[5,6,3,9,8,7,1,0,2,4,10] 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f52740a1890 0x7f52740a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): picked mon.noname-h con 0x7f52740a1890 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): picked mon.noname-d con 0x7f52740a1c60 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:37.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52740a2030 0x7f5274133660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): picked mon.noname-g con 0x7f52740a2030 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740a13c0 con 0x7f52740a2030 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f527409bdf0 con 0x7f52740a1890 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f527409cf90 con 0x7f52740a1c60 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.259+0000 7f527a528700 10 monclient(hunting): authenticate will time out at 1126.619912s 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56208/0 (socket says 172.21.15.12:56208) 2023-09-29T23:13:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 -- 172.21.15.12:0/1286366911 learned_addr learned my addr 172.21.15.12:0/1286366911 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): get_auth_request con 0x7f52740a1c60 auth_method 0 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279d27700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52740a2030 0x7f5274133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279526700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f52740a1890 0x7f52740a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279526700 10 monclient(hunting): get_auth_request con 0x7f52740a1890 auth_method 0 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279526700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279526700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279526700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279d27700 10 monclient(hunting): get_auth_request con 0x7f52740a2030 auth_method 0 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279d27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279d27700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5279d27700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 -- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f52740a1890 msgr2=0x7f52740a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f52740a1890 0x7f52740a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 -- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52740a2030 msgr2=0x7f5274133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52740a2030 0x7f5274133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient: _finish_hunting 0 2023-09-29T23:13:37.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 monclient: found mon.noname-d 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 -- 172.21.15.12:0/1286366911 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5274068ed0 con 0x7f52740a1c60 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5278d25700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52680099e0 tx=0x7f5268009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6bc4789630815425 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 1 -- 172.21.15.12:0/1286366911 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f526800bce0 con 0x7f52740a1c60 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: dump: 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:37.274 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 1 -- 172.21.15.12:0/1286366911 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526800be60 con 0x7f52740a1c60 2023-09-29T23:13:37.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f527a528700 5 monclient: authenticate success, global_id 4263 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f527a528700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 1 -- 172.21.15.12:0/1286366911 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f52680118a0 con 0x7f52740a1c60 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f527a528700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f527a528700 4 set_mon_vals no callback set 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.260+0000 7f5273fff700 10 monclient: dump: 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:37.278 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:37.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:37.280 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 10 monclient: shutdown 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 -- 172.21.15.12:0/1286366911 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 msgr2=0x7f5274133120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52680099e0 tx=0x7f5268009250 comp rx=0 tx=0).stop 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 -- 172.21.15.12:0/1286366911 shutdown_connections 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52740a2030 0x7f5274133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f52740a1890 0x7f52740a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 --2- 172.21.15.12:0/1286366911 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 -- 172.21.15.12:0/1286366911 >> 172.21.15.12:0/1286366911 conn(0x7f52740a2f80 msgr2=0x7f52740a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:37.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 -- 172.21.15.12:0/1286366911 shutdown_connections 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 1 -- 172.21.15.12:0/1286366911 wait complete. 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 10 monclient: build_initial_monmap 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.261+0000 7f527a528700 10 monclient: monmap: 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:37.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 Processor -- start 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 -- start start 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient: init 2023-09-29T23:13:37.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient: _add_conns ranks=[4,6,1,0,8,7,3,2,5] 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): picked mon.noname-c con 0x7f52740a1890 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): picked mon.noname-d con 0x7f52740a1c60 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52740a2030 0x7f5274133ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): picked mon.noname-f con 0x7f52740a2030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527409b700 con 0x7f52740a2030 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f527409b4d0 con 0x7f52740a1890 2023-09-29T23:13:37.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f527411d060 con 0x7f52740a1c60 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5278d25700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f527a528700 10 monclient(hunting): authenticate will time out at 1126.622953s 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37798/0 (socket says 172.21.15.12:37798) 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 1 -- 172.21.15.12:0/3583527713 learned_addr learned my addr 172.21.15.12:0/3583527713 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 10 monclient(hunting): get_auth_request con 0x7f52740a1890 auth_method 0 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5279526700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5278d25700 10 monclient(hunting): get_auth_request con 0x7f52740a1c60 auth_method 0 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5278d25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5278d25700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.262+0000 7f5278d25700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5278d25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279d27700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52740a2030 0x7f5274133ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279d27700 10 monclient(hunting): get_auth_request con 0x7f52740a2030 auth_method 0 2023-09-29T23:13:37.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279d27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279d27700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279d27700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 -- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 msgr2=0x7f5274133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 -- 172.21.15.12:0/3583527713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52740a2030 msgr2=0x7f5274133ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52740a2030 0x7f5274133ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient: _finish_hunting 0 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 monclient: found mon.noname-c 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:37.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 -- 172.21.15.12:0/3583527713 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680094f0 con 0x7f52740a1890 2023-09-29T23:13:37.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f5279526700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f526c00d890 tx=0x7f526c008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f526c007490 con 0x7f52740a1890 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: dump: 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:37.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:37.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c012a70 con 0x7f52740a1890 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 5 monclient: authenticate success, global_id 4288 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 10 monclient: _renew_subs 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527404c960 con 0x7f52740a1890 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f526c012be0 con 0x7f52740a1890 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52717fa700 10 monclient: dump: 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:37.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:37.292 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f52737fe700 4 set_mon_vals no callback set 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 10 monclient: _renew_subs 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.263+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527411ccf0 con 0x7f52740a1890 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f526c012d40 con 0x7f52740a1890 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f527a528700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f527a528700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f527a528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f52717fa700 1 --2- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 0x7f5260038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527411d340 con 0x7f52740a1890 2023-09-29T23:13:37.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3491+0+0 (secure 0 0 0) 0x7f526c047900 con 0x7f52740a1890 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f5278d25700 1 --2- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 0x7f5260038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.264+0000 7f5278d25700 10 monclient: get_auth_request con 0x7f5260036850 auth_method 0 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.265+0000 7f5278d25700 1 --2- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 0x7f5260038d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f526800bfd0 tx=0x7f5268017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.269+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f526c011340 con 0x7f52740a1890 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.269+0000 7f52717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:37.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.269+0000 7f52717fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:37.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.464+0000 7f527a528700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-29T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.464+0000 7f527a528700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-29T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.464+0000 7f527a528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:37.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:37.464+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5274070bc0 con 0x7f52740a1890 2023-09-29T23:13:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.126+0000 7f52717fa700 1 -- 172.21.15.12:0/3583527713 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f526c01db60 con 0x7f52740a1890 2023-09-29T23:13:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.126+0000 7f52717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-29T23:13:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.126+0000 7f52717fa700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.129+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 msgr2=0x7f5260038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.129+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 0x7f5260038d00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f526800bfd0 tx=0x7f5268017000 comp rx=0 tx=0).stop 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 10 monclient: shutdown 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 msgr2=0x7f527412fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f526c00d890 tx=0x7f526c008040 comp rx=0 tx=0).stop 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 shutdown_connections 2023-09-29T23:13:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> v2:172.21.15.153:6812/83257 conn(0x7f5260036850 0x7f5260038d00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52740a2030 0x7f5274133ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f52740a1890 0x7f527412fbd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 --2- 172.21.15.12:0/3583527713 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52740a1c60 0x7f5274133960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 >> 172.21.15.12:0/3583527713 conn(0x7f52740a2f80 msgr2=0x7f527409ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 shutdown_connections 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.130+0000 7f527a528700 1 -- 172.21.15.12:0/3583527713 wait complete. 2023-09-29T23:13:38.131 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T23:13:38.142 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-29T23:13:38.143 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-29T23:13:38.143 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-29T23:13:38.143 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-29T23:13:38.143 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 10 monclient: get_monmap_and_config 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 10 monclient: build_initial_monmap 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 10 monclient: monmap: 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:38.262895+0000 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:38.262895+0000 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:38.263 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:38.264 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:38.264 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:38.264 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 1 Processor -- start 2023-09-29T23:13:38.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 1 -- start start 2023-09-29T23:13:38.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.262+0000 7fd2e5f42700 10 monclient: init 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient: _add_conns ranks=[10,9,2,6,1,5,4,7,0,8,3] 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd2e00a1f90 0x7fd2e009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-f con 0x7fd2e00a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2e00a2360 0x7fd2e00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-a con 0x7fd2e00a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-b con 0x7fd2e0133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e009e350 con 0x7fd2e00a2360 2023-09-29T23:13:38.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e009ef70 con 0x7fd2e00a1f90 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e009da70 con 0x7fd2e0133740 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2e5f42700 10 monclient(hunting): authenticate will time out at 1127.623389s 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2e00a2360 0x7fd2e00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fd2e00a2360 0x7fd2e00a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52812/0 (socket says 172.21.15.12:52812) 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34688/0 (socket says 172.21.15.12:34688) 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 -- 172.21.15.12:0/637579596 learned_addr learned my addr 172.21.15.12:0/637579596 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:38.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): get_auth_request con 0x7fd2e00a2360 auth_method 0 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): get_auth_request con 0x7fd2e0133740 auth_method 0 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 1 --2- 172.21.15.12:0/637579596 >> v2:172.21.15.153:3300/0 conn(0x7fd2e00a1f90 0x7fd2e009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): get_auth_request con 0x7fd2e00a1f90 auth_method 0 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 -- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 msgr2=0x7fd2e0133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:38.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 -- 172.21.15.12:0/637579596 >> v2:172.21.15.153:3300/0 conn(0x7fd2e00a1f90 msgr2=0x7fd2e009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 --2- 172.21.15.12:0/637579596 >> v2:172.21.15.153:3300/0 conn(0x7fd2e00a1f90 0x7fd2e009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 monclient: found mon.noname-a 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2deffd700 1 -- 172.21.15.12:0/637579596 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0068ed0 con 0x7fd2e00a2360 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.263+0000 7fd2dffff700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2deffd700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e00a2360 0x7fd2e00a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0064b10 tx=0x7fd2d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff51ade9999edbac server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 1 -- 172.21.15.12:0/637579596 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd2d400bac0 con 0x7fd2e00a2360 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: dump: 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.278 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 1 -- 172.21.15.12:0/637579596 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d400bc40 con 0x7fd2e00a2360 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 5 monclient: authenticate success, global_id 4275 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:38.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 1 -- 172.21.15.12:0/637579596 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd2d400bdb0 con 0x7fd2e00a2360 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 4 set_mon_vals no callback set 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2de7fc700 10 monclient: dump: 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 10 monclient: shutdown 2023-09-29T23:13:38.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 -- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e00a2360 msgr2=0x7fd2e00a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e00a2360 0x7fd2e00a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0064b10 tx=0x7fd2d4009250 comp rx=0 tx=0).stop 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 -- 172.21.15.12:0/637579596 shutdown_connections 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd2e0133740 0x7fd2e0133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/637579596 >> v2:172.21.15.153:3300/0 conn(0x7fd2e00a1f90 0x7fd2e009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/637579596 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e00a2360 0x7fd2e00a27b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 -- 172.21.15.12:0/637579596 >> 172.21.15.12:0/637579596 conn(0x7fd2e00a3b50 msgr2=0x7fd2e00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 -- 172.21.15.12:0/637579596 shutdown_connections 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.264+0000 7fd2e5f42700 1 -- 172.21.15.12:0/637579596 wait complete. 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.265+0000 7fd2e5f42700 10 monclient: build_initial_monmap 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.265+0000 7fd2e5f42700 10 monclient: monmap: 2023-09-29T23:13:38.282 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:38.283 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.265+0000 7fd2e5f42700 1 Processor -- start 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.265+0000 7fd2e5f42700 1 -- start start 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.265+0000 7fd2e5f42700 10 monclient: init 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient: _add_conns ranks=[1,8,0,5,2,7,4,3,6] 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 0x7fd2e009c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-f con 0x7fd2e00a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd2e00a2360 0x7fd2e009c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-e con 0x7fd2e00a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:38.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): picked mon.noname-a con 0x7fd2e0133740 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e009ffc0 con 0x7fd2e0133740 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0130f10 con 0x7fd2e00a1f90 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e012eec0 con 0x7fd2e00a2360 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2e5f42700 10 monclient(hunting): authenticate will time out at 1127.626260s 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52824/0 (socket says 172.21.15.12:52824) 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 -- 172.21.15.12:0/654883295 learned_addr learned my addr 172.21.15.12:0/654883295 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): get_auth_request con 0x7fd2e0133740 auth_method 0 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd2e00a2360 0x7fd2e009c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): get_auth_request con 0x7fd2e00a2360 auth_method 0 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2deffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2df7fe700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 0x7fd2e009c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 -- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd2e00a2360 msgr2=0x7fd2e009c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd2e00a2360 0x7fd2e009c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 -- 172.21.15.12:0/654883295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 msgr2=0x7fd2e009c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 0x7fd2e009c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:38.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 monclient: found mon.noname-a 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 -- 172.21.15.12:0/654883295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8009a20 con 0x7fd2e0133740 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2df7fe700 10 monclient: get_auth_request con 0x7fd2e00a1f90 auth_method 0 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2df7fe700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 0x7fd2e009c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2dffff700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009fc0 tx=0x7fd2d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd2d8009d20 con 0x7fd2e0133740 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2c77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.266+0000 7fd2c77fe700 10 monclient: dump: 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.289 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d800f040 con 0x7fd2e0133740 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 5 monclient: authenticate success, global_id 4284 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _renew_subs 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d40094f0 con 0x7fd2e0133740 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd2d8014d80 con 0x7fd2e0133740 2023-09-29T23:13:38.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 10 monclient: dump: 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.291 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _renew_subs 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2ddffb700 4 set_mon_vals no callback set 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e004c960 con 0x7fd2e0133740 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd2d8003020 con 0x7fd2e0133740 2023-09-29T23:13:38.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0070bc0 con 0x7fd2e0133740 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.267+0000 7fd2c77fe700 1 --2- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 0x7fd2cc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.268+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd2d8047bc0 con 0x7fd2e0133740 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.268+0000 7fd2df7fe700 1 --2- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 0x7fd2cc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.268+0000 7fd2df7fe700 10 monclient: get_auth_request con 0x7fd2cc036880 auth_method 0 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.268+0000 7fd2df7fe700 1 --2- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 0x7fd2cc038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0006f00 tx=0x7fd2d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.272+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2d8012750 con 0x7fd2e0133740 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.273+0000 7fd2c77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.273+0000 7fd2c77fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:38.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.524+0000 7fd2e5f42700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-29T23:13:38.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.524+0000 7fd2e5f42700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-29T23:13:38.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.524+0000 7fd2e5f42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.524+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd2e0131ca0 con 0x7fd2e0133740 2023-09-29T23:13:38.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.525+0000 7fd2c77fe700 1 -- 172.21.15.12:0/654883295 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+82386 (secure 0 0 0) 0x7fd2d8018030 con 0x7fd2e0133740 2023-09-29T23:13:38.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.525+0000 7fd2c77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-29T23:13:38.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.525+0000 7fd2c77fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:38.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 msgr2=0x7fd2cc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 0x7fd2cc038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0006f00 tx=0x7fd2d0008040 comp rx=0 tx=0).stop 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 10 monclient: shutdown 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 msgr2=0x7fd2e009cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009fc0 tx=0x7fd2d800b040 comp rx=0 tx=0).stop 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 shutdown_connections 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> v2:172.21.15.153:6812/83257 conn(0x7fd2cc036880 0x7fd2cc038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2e0133740 0x7fd2e009cf00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2e00a1f90 0x7fd2e009c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.527+0000 7fd2e5f42700 1 --2- 172.21.15.12:0/654883295 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd2e00a2360 0x7fd2e009c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.528+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 >> 172.21.15.12:0/654883295 conn(0x7fd2e00a3b50 msgr2=0x7fd2e006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.528+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 shutdown_connections 2023-09-29T23:13:38.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.528+0000 7fd2e5f42700 1 -- 172.21.15.12:0/654883295 wait complete. 2023-09-29T23:13:38.540 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":83257}]},"active_addr":"172.21.15.153:6812/83257","active_change":"2023-09-29T23:13:32.716871+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":239825754}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3251435343}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":158545775}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1440526239}]}]} 2023-09-29T23:13:38.544 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-29T23:13:38.544 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-29T23:13:38.544 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:13:38.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 10 monclient: get_monmap_and_config 2023-09-29T23:13:38.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 10 monclient: build_initial_monmap 2023-09-29T23:13:38.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 10 monclient: monmap: 2023-09-29T23:13:38.663 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:38.663994+0000 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:38.663994+0000 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:38.664 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:38.665 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 1 Processor -- start 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 1 -- start start 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.663+0000 7fc389a25700 10 monclient: init 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient: _add_conns ranks=[6,10,3,0,4,9,2,8,1,7,5] 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-d con 0x7fc3840a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:38.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 0x7fc3840a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-f con 0x7fc3840a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc384133740 0x7fc384133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-g con 0x7fc384133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38409e350 con 0x7fc3840a2360 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38409ef70 con 0x7fc384133740 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc38409da70 con 0x7fc3840a1f90 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:38.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 10 monclient(hunting): authenticate will time out at 1128.024241s 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56222/0 (socket says 172.21.15.12:56222) 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 -- 172.21.15.12:0/2229415057 learned_addr learned my addr 172.21.15.12:0/2229415057 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): get_auth_request con 0x7fc3840a1f90 auth_method 0 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3837fe700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc384133740 0x7fc384133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3837fe700 10 monclient(hunting): get_auth_request con 0x7fc384133740 auth_method 0 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3837fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3827fc700 1 --2- 172.21.15.12:0/2229415057 >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 0x7fc3840a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 -- 172.21.15.12:0/2229415057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc384133740 msgr2=0x7fc384133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc384133740 0x7fc384133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 -- 172.21.15.12:0/2229415057 >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 msgr2=0x7fc3840a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 --2- 172.21.15.12:0/2229415057 >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 0x7fc3840a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 monclient: found mon.noname-d 2023-09-29T23:13:38.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 -- 172.21.15.12:0/2229415057 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384068ed0 con 0x7fc3840a1f90 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3827fc700 10 monclient: get_auth_request con 0x7fc3840a2360 auth_method 0 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc3827fc700 1 --2- 172.21.15.12:0/2229415057 >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 0x7fc3840a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc382ffd700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009a10 tx=0x7fc36c009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=78ceca1abd3400f8 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 1 -- 172.21.15.12:0/2229415057 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc36c00bc60 con 0x7fc3840a1f90 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 10 monclient: dump: 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.679 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc389a25700 5 monclient: authenticate success, global_id 4281 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.664+0000 7fc381ffb700 1 -- 172.21.15.12:0/2229415057 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c00bde0 con 0x7fc3840a1f90 2023-09-29T23:13:38.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc381ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 4 set_mon_vals no callback set 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc381ffb700 1 -- 172.21.15.12:0/2229415057 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc36c0118a0 con 0x7fc3840a1f90 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc381ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc381ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc381ffb700 10 monclient: dump: 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.681 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 10 monclient: shutdown 2023-09-29T23:13:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 -- 172.21.15.12:0/2229415057 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 msgr2=0x7fc38409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009a10 tx=0x7fc36c009250 comp rx=0 tx=0).stop 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 -- 172.21.15.12:0/2229415057 shutdown_connections 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc384133740 0x7fc384133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 --2- 172.21.15.12:0/2229415057 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc3840a1f90 0x7fc38409f5e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 --2- 172.21.15.12:0/2229415057 >> v2:172.21.15.153:3300/0 conn(0x7fc3840a2360 0x7fc3840a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 -- 172.21.15.12:0/2229415057 >> 172.21.15.12:0/2229415057 conn(0x7fc3840a3b50 msgr2=0x7fc3840a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 -- 172.21.15.12:0/2229415057 shutdown_connections 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.665+0000 7fc389a25700 1 -- 172.21.15.12:0/2229415057 wait complete. 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient: build_initial_monmap 2023-09-29T23:13:38.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient: monmap: 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 1 Processor -- start 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 1 -- start start 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient: init 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient: _add_conns ranks=[7,0,8,3,6,5,2,1,4] 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc3840a1f90 0x7fc384142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-i con 0x7fc3840a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.666+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-a con 0x7fc3840a2360 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fc384133740 0x7fc38412d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): picked mon.noname-e con 0x7fc384133740 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38409ffc0 con 0x7fc3840a2360 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc384130db0 con 0x7fc3840a1f90 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc384130ad0 con 0x7fc384133740 2023-09-29T23:13:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient(hunting): authenticate will time out at 1128.027184s 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52838/0 (socket says 172.21.15.12:52838) 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 -- 172.21.15.12:0/1969911310 learned_addr learned my addr 172.21.15.12:0/1969911310 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): get_auth_request con 0x7fc3840a2360 auth_method 0 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc3840a1f90 0x7fc384142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): get_auth_request con 0x7fc3840a1f90 auth_method 0 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fc384133740 0x7fc38412d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc382ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): get_auth_request con 0x7fc384133740 auth_method 0 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:38.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 -- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fc384133740 msgr2=0x7fc38412d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fc384133740 0x7fc38412d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 -- 172.21.15.12:0/1969911310 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc3840a1f90 msgr2=0x7fc384142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc3840a1f90 0x7fc384142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 monclient: found mon.noname-a 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:38.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 -- 172.21.15.12:0/1969911310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0094f0 con 0x7fc3840a2360 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3827fc700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc37400c880 tx=0x7fc374008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc374011a80 con 0x7fc3840a2360 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: dump: 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: _finish_auth 0 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374011c40 con 0x7fc3840a2360 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 5 monclient: authenticate success, global_id 4293 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient: _renew_subs 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38404c960 con 0x7fc3840a2360 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc374011db0 con 0x7fc3840a2360 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc37affd700 10 monclient: dump: 2023-09-29T23:13:38.692 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:38.693 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc3817fa700 4 set_mon_vals no callback set 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient: _renew_subs 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.667+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38412f030 con 0x7fc3840a2360 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.668+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc374011600 con 0x7fc3840a2360 2023-09-29T23:13:38.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.668+0000 7fc389a25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.668+0000 7fc389a25700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.668+0000 7fc389a25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.668+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384070bc0 con 0x7fc3840a2360 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.669+0000 7fc37affd700 1 --2- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 0x7fc364038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.669+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc374047140 con 0x7fc3840a2360 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.669+0000 7fc382ffd700 1 --2- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 0x7fc364038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.669+0000 7fc382ffd700 10 monclient: get_auth_request con 0x7fc364036850 auth_method 0 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.669+0000 7fc382ffd700 1 --2- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 0x7fc364038d00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009fd0 tx=0x7fc36c017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.673+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3740118b0 con 0x7fc3840a2360 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.673+0000 7fc37affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:38.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.673+0000 7fc37affd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:38.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.867+0000 7fc389a25700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:38.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.867+0000 7fc389a25700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:38.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.867+0000 7fc389a25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:38.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.867+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc384066790 con 0x7fc3840a2360 2023-09-29T23:13:38.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.868+0000 7fc37affd700 1 -- 172.21.15.12:0/1969911310 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+9991 (secure 0 0 0) 0x7fc37401f030 con 0x7fc3840a2360 2023-09-29T23:13:38.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.868+0000 7fc37affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:38.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.868+0000 7fc37affd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:38.869 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:38.869 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:38.117969+0000","last_up_change":"2023-09-29T23:13:31.402198+0000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:13:32.778797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:13:36.755733+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":83761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":83761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":83761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":83761}]},"public_addr":"172.21.15.12:6802/83761","cluster_addr":"172.21.15.12:6804/83761","heartbeat_back_addr":"172.21.15.12:6807/83761","heartbeat_front_addr":"172.21.15.12:6806/83761","state":["exists","up"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":83762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6801","nonce":83762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6805","nonce":83762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":83762}]},"public_addr":"172.21.15.12:6800/83762","cluster_addr":"172.21.15.12:6801/83762","heartbeat_back_addr":"172.21.15.12:6805/83762","heartbeat_front_addr":"172.21.15.12:6803/83762","state":["exists","up"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":83763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":83763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":83763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":83763}]},"public_addr":"172.21.15.12:6808/83763","cluster_addr":"172.21.15.12:6809/83763","heartbeat_back_addr":"172.21.15.12:6811/83763","heartbeat_front_addr":"172.21.15.12:6810/83763","state":["exists","up"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":83793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":83793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":83793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":83793}]},"public_addr":"172.21.15.153:6800/83793","cluster_addr":"172.21.15.153:6801/83793","heartbeat_back_addr":"172.21.15.153:6804/83793","heartbeat_front_addr":"172.21.15.153:6802/83793","state":["exists","up"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":83794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":83794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":83794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":83794}]},"public_addr":"172.21.15.153:6803/83794","cluster_addr":"172.21.15.153:6805/83794","heartbeat_back_addr":"172.21.15.153:6807/83794","heartbeat_front_addr":"172.21.15.153:6806/83794","state":["exists","up"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":83795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":83795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":83795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":83795}]},"public_addr":"172.21.15.153:6808/83795","cluster_addr":"172.21.15.153:6809/83795","heartbeat_back_addr":"172.21.15.153:6811/83795","heartbeat_front_addr":"172.21.15.153:6810/83795","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.551171+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.484006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.491551+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.520272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.501043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.509993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:13:38.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 msgr2=0x7fc364038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 0x7fc364038d00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009fd0 tx=0x7fc36c017000 comp rx=0 tx=0).stop 2023-09-29T23:13:38.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 10 monclient: shutdown 2023-09-29T23:13:38.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:38.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 msgr2=0x7fc3841328a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:38.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc37400c880 tx=0x7fc374008040 comp rx=0 tx=0).stop 2023-09-29T23:13:38.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 shutdown_connections 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> v2:172.21.15.153:6812/83257 conn(0x7fc364036850 0x7fc364038d00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fc384133740 0x7fc38412d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc3840a1f90 0x7fc384142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 --2- 172.21.15.12:0/1969911310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3840a2360 0x7fc3841328a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 >> 172.21.15.12:0/1969911310 conn(0x7fc3840a3b50 msgr2=0x7fc38406f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 shutdown_connections 2023-09-29T23:13:38.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.870+0000 7fc389a25700 1 -- 172.21.15.12:0/1969911310 wait complete. 2023-09-29T23:13:38.882 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-29T23:13:38.882 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:13:39.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.999+0000 7fa97f59e700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.999+0000 7fa97f59e700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:38.999+0000 7fa97f59e700 10 monclient: monmap: 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.000661+0000 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.000661+0000 2023-09-29T23:13:39.001 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.002 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.003 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.004 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.000+0000 7fa97f59e700 1 Processor -- start 2023-09-29T23:13:39.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.000+0000 7fa97f59e700 1 -- start start 2023-09-29T23:13:39.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.000+0000 7fa97f59e700 10 monclient: init 2023-09-29T23:13:39.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.000+0000 7fa97f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.000+0000 7fa97f59e700 10 monclient: _add_conns ranks=[5,2,7,3,4,0,10,6,8,9,1] 2023-09-29T23:13:39.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa980133130 0x7fa9800a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-h con 0x7fa980133130 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa980133500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa98012fc20 0x7fa980130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-i con 0x7fa98012fc20 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9800a27a0 con 0x7fa980133500 2023-09-29T23:13:39.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa98009adf0 con 0x7fa980133130 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa98009bd60 con 0x7fa98012fc20 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient(hunting): authenticate will time out at 1128.361069s 2023-09-29T23:13:39.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34694/0 (socket says 172.21.15.12:34694) 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 -- 172.21.15.12:0/4061639293 learned_addr learned my addr 172.21.15.12:0/4061639293 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): get_auth_request con 0x7fa980133500 auth_method 0 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97e59c700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa980133130 0x7fa9800a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97ed9d700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa98012fc20 0x7fa980130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97ed9d700 10 monclient(hunting): get_auth_request con 0x7fa98012fc20 auth_method 0 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97e59c700 10 monclient(hunting): get_auth_request con 0x7fa980133130 auth_method 0 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 -- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa98012fc20 msgr2=0x7fa980130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa98012fc20 0x7fa980130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 -- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa980133130 msgr2=0x7fa9800a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa980133130 0x7fa9800a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 monclient: found mon.noname-b 2023-09-29T23:13:39.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:39.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 -- 172.21.15.12:0/4061639293 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa980068ed0 con 0x7fa980133500 2023-09-29T23:13:39.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97dd9b700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa968009a10 tx=0x7fa968009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a9e7620e841b78e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 1 -- 172.21.15.12:0/4061639293 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa96800bca0 con 0x7fa980133500 2023-09-29T23:13:39.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:39.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: dump: 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.031 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.032 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 5 monclient: authenticate success, global_id 4205 2023-09-29T23:13:39.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 1 -- 172.21.15.12:0/4061639293 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96800be20 con 0x7fa980133500 2023-09-29T23:13:39.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97f59e700 4 set_mon_vals no callback set 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 1 -- 172.21.15.12:0/4061639293 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa9680118a0 con 0x7fa980133500 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.001+0000 7fa97d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97d59a700 10 monclient: dump: 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.036 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 10 monclient: shutdown 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 -- 172.21.15.12:0/4061639293 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 msgr2=0x7fa980133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa968009a10 tx=0x7fa968009250 comp rx=0 tx=0).stop 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 -- 172.21.15.12:0/4061639293 shutdown_connections 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa98012fc20 0x7fa980130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa980133130 0x7fa9800a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 --2- 172.21.15.12:0/4061639293 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa980133500 0x7fa980133950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 -- 172.21.15.12:0/4061639293 >> 172.21.15.12:0/4061639293 conn(0x7fa9800a6490 msgr2=0x7fa9800a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 -- 172.21.15.12:0/4061639293 shutdown_connections 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.002+0000 7fa97f59e700 1 -- 172.21.15.12:0/4061639293 wait complete. 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient: monmap: 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.039 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.040 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 1 Processor -- start 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 1 -- start start 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient: init 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient: _add_conns ranks=[6,0,3,2,1,8,7,5,4] 2023-09-29T23:13:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-d con 0x7fa98012fc20 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa980133130 0x7fa98006da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.003+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa980133130 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa980133500 0x7fa98006df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): picked mon.noname-g con 0x7fa980133500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98011de30 con 0x7fa980133130 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98011e360 con 0x7fa980133500 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa98011cb10 con 0x7fa98012fc20 2023-09-29T23:13:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient(hunting): authenticate will time out at 1128.364057s 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa980133130 0x7fa98006da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56234/0 (socket says 172.21.15.12:56234) 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 -- 172.21.15.12:0/629625310 learned_addr learned my addr 172.21.15.12:0/629625310 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): get_auth_request con 0x7fa98012fc20 auth_method 0 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): get_auth_request con 0x7fa980133130 auth_method 0 2023-09-29T23:13:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97ed9d700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa980133500 0x7fa98006df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97ed9d700 10 monclient(hunting): get_auth_request con 0x7fa980133500 auth_method 0 2023-09-29T23:13:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 -- 172.21.15.12:0/629625310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa980133500 msgr2=0x7fa98006df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa980133500 0x7fa98006df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 -- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa980133130 msgr2=0x7fa98006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa980133130 0x7fa98006da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 monclient: found mon.noname-d 2023-09-29T23:13:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 -- 172.21.15.12:0/629625310 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680094f0 con 0x7fa98012fc20 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97e59c700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa97800d890 tx=0x7fa978008040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa978007470 con 0x7fa98012fc20 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: dump: 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.046 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.047 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa978012a70 con 0x7fa98012fc20 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 5 monclient: authenticate success, global_id 4290 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient: _renew_subs 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98004c960 con 0x7fa98012fc20 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa978012be0 con 0x7fa98012fc20 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa9767fc700 10 monclient: dump: 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.049 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97cd99700 4 set_mon_vals no callback set 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient: _renew_subs 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.004+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98011d800 con 0x7fa98012fc20 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa978012d40 con 0x7fa98012fc20 2023-09-29T23:13:39.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa97f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa97f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa97f59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa980070bc0 con 0x7fa98012fc20 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa9767fc700 1 --2- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 0x7fa960038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.005+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa978047990 con 0x7fa98012fc20 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.006+0000 7fa97dd9b700 1 --2- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 0x7fa960038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.006+0000 7fa97dd9b700 10 monclient: get_auth_request con 0x7fa960036850 auth_method 0 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.006+0000 7fa97dd9b700 1 --2- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 0x7fa960038d00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa968009fd0 tx=0x7fa968017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.010+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9780125f0 con 0x7fa98012fc20 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.010+0000 7fa9767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.010+0000 7fa9767fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa97f59e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:39.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa97f59e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:39.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa97f59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa98009b9f0 con 0x7fa98012fc20 2023-09-29T23:13:39.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa9767fc700 1 -- 172.21.15.12:0/629625310 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+9978 (secure 0 0 0) 0x7fa978011b30 con 0x7fa98012fc20 2023-09-29T23:13:39.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa9767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:13:39.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.210+0000 7fa9767fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:39.211 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:39.211 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:39.119641+0000","last_up_change":"2023-09-29T23:13:31.402198+0000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:13:32.778797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:13:36.755733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":83761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":83761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":83761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":83761}]},"public_addr":"172.21.15.12:6802/83761","cluster_addr":"172.21.15.12:6804/83761","heartbeat_back_addr":"172.21.15.12:6807/83761","heartbeat_front_addr":"172.21.15.12:6806/83761","state":["exists","up"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":83762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6801","nonce":83762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6805","nonce":83762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":83762}]},"public_addr":"172.21.15.12:6800/83762","cluster_addr":"172.21.15.12:6801/83762","heartbeat_back_addr":"172.21.15.12:6805/83762","heartbeat_front_addr":"172.21.15.12:6803/83762","state":["exists","up"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":83763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":83763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":83763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":83763}]},"public_addr":"172.21.15.12:6808/83763","cluster_addr":"172.21.15.12:6809/83763","heartbeat_back_addr":"172.21.15.12:6811/83763","heartbeat_front_addr":"172.21.15.12:6810/83763","state":["exists","up"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":83793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":83793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":83793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":83793}]},"public_addr":"172.21.15.153:6800/83793","cluster_addr":"172.21.15.153:6801/83793","heartbeat_back_addr":"172.21.15.153:6804/83793","heartbeat_front_addr":"172.21.15.153:6802/83793","state":["exists","up"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":83794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":83794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":83794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":83794}]},"public_addr":"172.21.15.153:6803/83794","cluster_addr":"172.21.15.153:6805/83794","heartbeat_back_addr":"172.21.15.153:6807/83794","heartbeat_front_addr":"172.21.15.153:6806/83794","state":["exists","up"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":83795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":83795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":83795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":83795}]},"public_addr":"172.21.15.153:6808/83795","cluster_addr":"172.21.15.153:6809/83795","heartbeat_back_addr":"172.21.15.153:6811/83795","heartbeat_front_addr":"172.21.15.153:6810/83795","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.551171+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.484006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.491551+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.520272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.501043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.509993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 msgr2=0x7fa960038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 0x7fa960038d00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa968009fd0 tx=0x7fa968017000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 10 monclient: shutdown 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 msgr2=0x7fa98006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa97800d890 tx=0x7fa978008040 comp rx=0 tx=0).stop 2023-09-29T23:13:39.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.212+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 shutdown_connections 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> v2:172.21.15.153:6812/83257 conn(0x7fa960036850 0x7fa960038d00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa980133500 0x7fa98006df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fa98012fc20 0x7fa98006d500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 --2- 172.21.15.12:0/629625310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa980133130 0x7fa98006da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 >> 172.21.15.12:0/629625310 conn(0x7fa9800a6490 msgr2=0x7fa9801349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 shutdown_connections 2023-09-29T23:13:39.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.213+0000 7fa97f59e700 1 -- 172.21.15.12:0/629625310 wait complete. 2023-09-29T23:13:39.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-29T23:13:39.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-29T23:13:39.226 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-29T23:13:39.226 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-29T23:13:39.226 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-29T23:13:39.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-29T23:13:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.364+0000 7f48c6380700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.364+0000 7f48c6380700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.364+0000 7f48c6380700 10 monclient: monmap: 2023-09-29T23:13:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.365652+0000 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.365652+0000 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.366 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f1c28f2f700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f1c28f2f700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f1c28f2f700 10 monclient: monmap: 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.368 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.366280+0000 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.366280+0000 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.369 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f48c6380700 1 Processor -- start 2023-09-29T23:13:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f48c6380700 1 -- start start 2023-09-29T23:13:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.365+0000 7f48c6380700 10 monclient: init 2023-09-29T23:13:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f1c28f2f700 1 Processor -- start 2023-09-29T23:13:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f1c28f2f700 1 -- start start 2023-09-29T23:13:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f1c28f2f700 10 monclient: init 2023-09-29T23:13:39.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient: _add_conns ranks=[0,6,3,4,5,2,1,8,10,9,7] 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48c006ba40 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 0x7f48c006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-d con 0x7f48c006be10 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-g con 0x7f48c006c720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c0068ed0 con 0x7f48c006ba40 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c006d990 con 0x7f48c006c720 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f48c006dec0 con 0x7f48c006be10 2023-09-29T23:13:39.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c6380700 10 monclient(hunting): authenticate will time out at 1128.726639s 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.366+0000 7f48c491d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.12:50466/0 (socket says 172.21.15.12:50466) 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 -- 172.21.15.12:0/3602644882 learned_addr learned my addr 172.21.15.12:0/3602644882 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): get_auth_request con 0x7f48c006c720 auth_method 0 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient: _add_conns ranks=[4,7,9,1,2,6,10,8,5,0,3] 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-c con 0x7f1c2412f660 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c2406eb60 0x7f1c2406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-i con 0x7f1c2406eb60 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 0x7f1c2409da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-a con 0x7f1c2409d5a0 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2406bcf0 con 0x7f1c2409d5a0 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2406d090 con 0x7f1c2412f660 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2412fa30 con 0x7f1c2406eb60 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c28f2f700 10 monclient(hunting): authenticate will time out at 1128.727247s 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48bf7fe700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 0x7f48c006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 -- 172.21.15.12:0/3602644882 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 msgr2=0x7f48c006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 0x7f48c006c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 -- 172.21.15.12:0/3602644882 >> v1:172.21.15.12:6789/0 conn(0x7f48c006ba40 legacy=0x7f48c006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 monclient: found mon.noname-g 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.153:3301/0 2023-09-29T23:13:39.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48c491d700 1 -- 172.21.15.12:0/3602644882 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c004c960 con 0x7f48c006c720 2023-09-29T23:13:39.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48bf7fe700 10 monclient: get_auth_request con 0x7f48c006be10 auth_method 0 2023-09-29T23:13:39.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f48bf7fe700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 0x7f48c006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37810/0 (socket says 172.21.15.12:37810) 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 1 -- 172.21.15.12:0/2263872981 learned_addr learned my addr 172.21.15.12:0/2263872981 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): get_auth_request con 0x7f1c2412f660 auth_method 0 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 1 --2- 172.21.15.12:0/2263872981 >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 0x7f1c2409da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): get_auth_request con 0x7f1c2409d5a0 auth_method 0 2023-09-29T23:13:39.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.367+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 -- 172.21.15.12:0/2263872981 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c2406eb60 msgr2=0x7f1c2406ef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 --2- 172.21.15.12:0/2263872981 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c2406eb60 0x7f1c2406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 -- 172.21.15.12:0/2263872981 >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 msgr2=0x7f1c2409da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 --2- 172.21.15.12:0/2263872981 >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 0x7f1c2409da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 -- 172.21.15.12:0/2263872981 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24068ed0 con 0x7f1c2412f660 2023-09-29T23:13:39.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c22d9d700 1 --2- 172.21.15.12:0/2263872981 >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 0x7f1c2409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48c491d700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009a10 tx=0x7f48b0009250 comp rx=0 tx=0).ready entity=mon.3 client_cookie=46a06b13812b3d2e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 1 -- 172.21.15.12:0/3602644882 <== mon.3 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48b000bc60 con 0x7f48c006c720 2023-09-29T23:13:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2259c700 1 --2- 172.21.15.12:0/2263872981 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bc50 tx=0x7f1c1400b4f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4151ce213e21c223 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: dump: 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.384 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.386 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 1 -- 172.21.15.12:0/3602644882 <== mon.3 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000bde0 con 0x7f48c006c720 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 1 -- 172.21.15.12:0/3602644882 <== mon.3 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48b00118a0 con 0x7f48c006c720 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48beffd700 10 monclient: dump: 2023-09-29T23:13:39.387 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.388 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48c6380700 5 monclient: authenticate success, global_id 4224 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48c6380700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f48c6380700 4 set_mon_vals no callback set 2023-09-29T23:13:39.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 1 -- 172.21.15.12:0/2263872981 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c1400dbf0 con 0x7f1c2412f660 2023-09-29T23:13:39.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: dump: 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.391 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 1 -- 172.21.15.12:0/2263872981 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1400dd70 con 0x7f1c2412f660 2023-09-29T23:13:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 1 -- 172.21.15.12:0/2263872981 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c14019d80 con 0x7f1c2412f660 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c2159a700 10 monclient: dump: 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.393 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.394 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c28f2f700 5 monclient: authenticate success, global_id 4297 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c28f2f700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.368+0000 7f1c28f2f700 4 set_mon_vals no callback set 2023-09-29T23:13:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 10 monclient: shutdown 2023-09-29T23:13:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 -- 172.21.15.12:0/3602644882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 msgr2=0x7f48c01373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009a10 tx=0x7f48b0009250 comp rx=0 tx=0).stop 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 -- 172.21.15.12:0/3602644882 shutdown_connections 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c006c720 0x7f48c01373c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 --2- 172.21.15.12:0/3602644882 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006be10 0x7f48c006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 -- 172.21.15.12:0/3602644882 >> 172.21.15.12:0/3602644882 conn(0x7f48c00a2bb0 msgr2=0x7f48c00a2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 -- 172.21.15.12:0/3602644882 shutdown_connections 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f48c6380700 1 -- 172.21.15.12:0/3602644882 wait complete. 2023-09-29T23:13:39.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f1c28f2f700 10 monclient: shutdown 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f1c28f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f1c28f2f700 1 -- 172.21.15.12:0/2263872981 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 msgr2=0x7f1c2412fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.369+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/2263872981 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bc50 tx=0x7f1c1400b4f0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 -- 172.21.15.12:0/2263872981 shutdown_connections 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/2263872981 >> v2:172.21.15.12:3300/0 conn(0x7f1c2409d5a0 0x7f1c2409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/2263872981 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1c2412f660 0x7f1c2412fca0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/2263872981 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c2406eb60 0x7f1c2406ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 -- 172.21.15.12:0/2263872981 >> 172.21.15.12:0/2263872981 conn(0x7f1c2406a370 msgr2=0x7f1c240a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f48c6380700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 -- 172.21.15.12:0/2263872981 shutdown_connections 2023-09-29T23:13:39.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 1 -- 172.21.15.12:0/2263872981 wait complete. 2023-09-29T23:13:39.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f48c6380700 10 monclient: monmap: 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.400 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f48c6380700 1 Processor -- start 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f1c28f2f700 10 monclient: monmap: 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.402 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f48c6380700 1 -- start start 2023-09-29T23:13:39.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.370+0000 7f48c6380700 10 monclient: init 2023-09-29T23:13:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 Processor -- start 2023-09-29T23:13:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 -- start start 2023-09-29T23:13:39.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient: init 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient: _add_conns ranks=[4,0,6,1,3,8,2,5,7] 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48c006ba40 0x7f48c0142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-c con 0x7f48c006ba40 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48c006be10 0x7f48c0138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-a con 0x7f48c006be10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006c720 0x7f48c011b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): picked mon.noname-d con 0x7f48c006c720 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c011f370 con 0x7f48c006be10 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f48c011f090 con 0x7f48c006ba40 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f48c011f650 con 0x7f48c006c720 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c6380700 10 monclient(hunting): authenticate will time out at 1128.731514s 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006c720 0x7f48c011b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006c720 0x7f48c011b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56254/0 (socket says 172.21.15.12:56254) 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 1 -- 172.21.15.12:0/3330922158 learned_addr learned my addr 172.21.15.12:0/3330922158 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): get_auth_request con 0x7f48c006c720 auth_method 0 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 1 --2- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48c006ba40 0x7f48c0142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): get_auth_request con 0x7f48c006ba40 auth_method 0 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f48c491d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 -- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006c720 msgr2=0x7f48c011b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 --2- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f48c006c720 0x7f48c011b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 -- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48c006be10 msgr2=0x7f48c0138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 --2- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48c006be10 0x7f48c0138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 -- 172.21.15.12:0/3330922158 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00094f0 con 0x7f48c006ba40 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f48bffff700 1 --2- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48c006ba40 0x7f48c0142250 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f48b800bb00 tx=0x7f48b8009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient: _add_conns ranks=[2,0,5,4,3,8,1,7,6] 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 0x7f1c24142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-b con 0x7f1c2406eb60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2409d5a0 0x7f1c24142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-a con 0x7f1c2409d5a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c2412f660 0x7f1c2409b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): picked mon.noname-h con 0x7f1c2412f660 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241205b0 con 0x7f1c2409d5a0 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2406c5b0 con 0x7f1c2406eb60 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.371+0000 7f1c28f2f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2406c170 con 0x7f1c2412f660 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c28f2f700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c28f2f700 10 monclient(hunting): authenticate will time out at 1128.732017s 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 0x7f1c24142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 0x7f1c24142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34704/0 (socket says 172.21.15.12:34704) 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 -- 172.21.15.12:0/1896347256 learned_addr learned my addr 172.21.15.12:0/1896347256 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): get_auth_request con 0x7f1c2406eb60 auth_method 0 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c21d9b700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2409d5a0 0x7f1c24142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c21d9b700 10 monclient(hunting): get_auth_request con 0x7f1c2409d5a0 auth_method 0 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c21d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c21d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c21d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c2412f660 0x7f1c2409b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): get_auth_request con 0x7f1c2412f660 auth_method 0 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c22d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 -- 172.21.15.12:0/1896347256 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c2412f660 msgr2=0x7f1c2409b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c2412f660 0x7f1c2409b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 -- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2409d5a0 msgr2=0x7f1c24142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c2409d5a0 0x7f1c24142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 monclient: found mon.noname-b 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.372+0000 7f1c2259c700 1 -- 172.21.15.12:0/1896347256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c009a20 con 0x7f1c2406eb60 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48b800ccc0 con 0x7f48c006ba40 2023-09-29T23:13:39.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c2259c700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 0x7f1c24142a50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c2406d1d0 tx=0x7f1c1401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: dump: 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.415 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b800ce80 con 0x7f48c006ba40 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48b8014890 con 0x7f48c006ba40 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48a7fff700 10 monclient: dump: 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.417 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.418 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48be7fc700 4 set_mon_vals no callback set 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 5 monclient: authenticate success, global_id 4306 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 10 monclient: _renew_subs 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c0142790 con 0x7f48c006ba40 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 10 monclient: _renew_subs 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c0143020 con 0x7f48c006ba40 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c14010b00 con 0x7f1c2406eb60 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: dump: 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.420 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c14010ca0 con 0x7f1c2406eb60 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c14010e10 con 0x7f1c2406eb60 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c127fc700 10 monclient: dump: 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.373+0000 7f1c20d99700 4 set_mon_vals no callback set 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 5 monclient: authenticate success, global_id 4214 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 10 monclient: _renew_subs 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1400b790 con 0x7f1c2406eb60 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 10 monclient: _renew_subs 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:39.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2404c960 con 0x7f1c2406eb60 2023-09-29T23:13:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f48c6380700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48c011f200 con 0x7f48c006ba40 2023-09-29T23:13:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.374+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f48b8014410 con 0x7f48c006ba40 2023-09-29T23:13:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.153:6812/83257 conn(0x7f48ac036850 0x7f48ac038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f48b8047240 con 0x7f48c006ba40 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.12:6802/83761 conn(0x7f48ac03c2f0 0x7f48ac03e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 --> v2:172.21.15.12:6802/83761 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48ac03ed70 con 0x7f48ac03c2f0 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48b80146c0 con 0x7f48c006ba40 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48a7fff700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48bf7fe700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.153:6812/83257 conn(0x7f48ac036850 0x7f48ac038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48c491d700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.12:6802/83761 conn(0x7f48ac03c2f0 0x7f48ac03e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48c491d700 10 monclient: get_auth_request con 0x7f48ac03c2f0 auth_method 0 2023-09-29T23:13:39.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f1c28f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:39.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c240a1fb0 con 0x7f1c2406eb60 2023-09-29T23:13:39.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.375+0000 7f48bf7fe700 10 monclient: get_auth_request con 0x7f48ac036850 auth_method 0 2023-09-29T23:13:39.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1c14010680 con 0x7f1c2406eb60 2023-09-29T23:13:39.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f48c491d700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.12:6802/83761 conn(0x7f48ac03c2f0 0x7f48ac03e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f48bf7fe700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.153:6812/83257 conn(0x7f48ac036850 0x7f48ac038d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48c00a1050 tx=0x7f48b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.153:6812/83257 conn(0x7f1c04036850 0x7f1c04038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1c14014070 con 0x7f1c2406eb60 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c21d9b700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.153:6812/83257 conn(0x7f1c04036850 0x7f1c04038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c21d9b700 10 monclient: get_auth_request con 0x7f1c04036850 auth_method 0 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.12:6808/83763 conn(0x7f1c0403c2f0 0x7f1c0403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 --> v2:172.21.15.12:6808/83763 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c0403ed70 con 0x7f1c0403c2f0 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c14049120 con 0x7f1c2406eb60 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c127fc700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c22d9d700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.12:6808/83763 conn(0x7f1c0403c2f0 0x7f1c0403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.376+0000 7f1c22d9d700 10 monclient: get_auth_request con 0x7f1c0403c2f0 auth_method 0 2023-09-29T23:13:39.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.377+0000 7f1c21d9b700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.153:6812/83257 conn(0x7f1c04036850 0x7f1c04038d00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1c18006fd0 tx=0x7f1c18008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.377+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== osd.0 v2:172.21.15.12:6802/83761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f48ac03ed70 con 0x7f48ac03c2f0 2023-09-29T23:13:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.377+0000 7f1c22d9d700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.12:6808/83763 conn(0x7f1c0403c2f0 0x7f1c0403e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.378+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== osd.2 v2:172.21.15.12:6808/83763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f1c0403ed70 con 0x7f1c0403c2f0 2023-09-29T23:13:39.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.404+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 --> v2:172.21.15.12:6802/83761 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f48c011f4e0 con 0x7f48ac03c2f0 2023-09-29T23:13:39.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.404+0000 7f48a7fff700 1 -- 172.21.15.12:0/3330922158 <== osd.0 v2:172.21.15.12:6802/83761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f48c011f4e0 con 0x7f48ac03c2f0 2023-09-29T23:13:39.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 >> v2:172.21.15.12:6802/83761 conn(0x7f48ac03c2f0 msgr2=0x7f48ac03e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.12:6802/83761 conn(0x7f48ac03c2f0 0x7f48ac03e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 >> v2:172.21.15.153:6812/83257 conn(0x7f48ac036850 msgr2=0x7f48ac038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 --2- 172.21.15.12:0/3330922158 >> v2:172.21.15.153:6812/83257 conn(0x7f48ac036850 0x7f48ac038d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48c00a1050 tx=0x7f48b4008040 comp rx=0 tx=0).stop 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 10 monclient: shutdown 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48c006ba40 msgr2=0x7f48c0142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c6380700 1 --2- 172.21.15.12:0/3330922158 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48c006ba40 0x7f48c0142250 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f48b800bb00 tx=0x7f48b8009780 comp rx=0 tx=0).stop 2023-09-29T23:13:39.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.405+0000 7f48c491d700 1 -- 172.21.15.12:0/3330922158 reap_dead start 2023-09-29T23:13:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.406+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 --> v2:172.21.15.12:6808/83763 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c24070bc0 con 0x7f1c0403c2f0 2023-09-29T23:13:39.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.406+0000 7f1c127fc700 1 -- 172.21.15.12:0/1896347256 <== osd.2 v2:172.21.15.12:6808/83763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c24070bc0 con 0x7f1c0403c2f0 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 >> v2:172.21.15.12:6808/83763 conn(0x7f1c0403c2f0 msgr2=0x7f1c0403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.12:6808/83763 conn(0x7f1c0403c2f0 0x7f1c0403e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 >> v2:172.21.15.153:6812/83257 conn(0x7f1c04036850 msgr2=0x7f1c04038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/1896347256 >> v2:172.21.15.153:6812/83257 conn(0x7f1c04036850 0x7f1c04038d00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1c18006fd0 tx=0x7f1c18008040 comp rx=0 tx=0).stop 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 10 monclient: shutdown 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 msgr2=0x7f1c24142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 --2- 172.21.15.12:0/1896347256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1c2406eb60 0x7f1c24142a50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c2406d1d0 tx=0x7f1c1401b000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c22d9d700 1 -- 172.21.15.12:0/1896347256 reap_dead start 2023-09-29T23:13:39.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 shutdown_connections 2023-09-29T23:13:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 >> 172.21.15.12:0/1896347256 conn(0x7f1c2406a370 msgr2=0x7f1c2409fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 shutdown_connections 2023-09-29T23:13:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.407+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 >> 172.21.15.12:0/3330922158 conn(0x7f48c00a2bb0 msgr2=0x7f48c009c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.408+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 shutdown_connections 2023-09-29T23:13:39.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.408+0000 7f48c6380700 1 -- 172.21.15.12:0/3330922158 wait complete. 2023-09-29T23:13:39.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.409+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 shutdown_connections 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.410+0000 7f9f6d33f700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.410+0000 7f9f6d33f700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.410+0000 7f9f6d33f700 10 monclient: monmap: 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.410968+0000 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.410968+0000 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.441 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.410+0000 7f9f6d33f700 1 Processor -- start 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 -- start start 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient: init 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient: _add_conns ranks=[5,3,2,6,0,4,7,1,10,8,9] 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 0x7f9f6809d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-h con 0x7f9f6809d5a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f6809cba0 0x7f9f6809deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-g con 0x7f9f6809cba0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 0x7f9f681333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-b con 0x7f9f6809adf0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6809bed0 con 0x7f9f6809adf0 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6809bbf0 con 0x7f9f6809cba0 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6809e3f0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.411+0000 7f9f6d33f700 10 monclient(hunting): authenticate will time out at 1128.771831s 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 0x7f9f681333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 0x7f9f6809d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f6809cba0 0x7f9f6809deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 0x7f9f681333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34716/0 (socket says 172.21.15.12:34716) 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 1 -- 172.21.15.12:0/803121029 learned_addr learned my addr 172.21.15.12:0/803121029 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): get_auth_request con 0x7f9f6809adf0 auth_method 0 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): get_auth_request con 0x7f9f6809d5a0 auth_method 0 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): get_auth_request con 0x7f9f6809cba0 auth_method 0 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 -- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f6809cba0 msgr2=0x7f9f6809deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f6809cba0 0x7f9f6809deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 -- 172.21.15.12:0/803121029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 msgr2=0x7f9f681333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 0x7f9f681333b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 monclient: found mon.noname-h 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 -- 172.21.15.12:0/803121029 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f68068ed0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 0x7f9f6809d970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009910 tx=0x7f9f500091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=6aa3ee1eb63e25f2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 1 -- 172.21.15.12:0/803121029 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f50011ab0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: dump: 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.447 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.448 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f6d33f700 5 monclient: authenticate success, global_id 4226 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f6d33f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 1 -- 172.21.15.12:0/803121029 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f50011c30 con 0x7f9f6809d5a0 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f6d33f700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 1 -- 172.21.15.12:0/803121029 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f50011da0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f6d33f700 4 set_mon_vals no callback set 2023-09-29T23:13:39.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.412+0000 7f9f65ffb700 10 monclient: dump: 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.450 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 10 monclient: shutdown 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 -- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 msgr2=0x7f9f6809d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 0x7f9f6809d970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009910 tx=0x7f9f500091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 -- 172.21.15.12:0/803121029 shutdown_connections 2023-09-29T23:13:39.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9f6809d5a0 0x7f9f6809d970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f6809adf0 0x7f9f681333b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/803121029 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f6809cba0 0x7f9f6809deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 -- 172.21.15.12:0/803121029 >> 172.21.15.12:0/803121029 conn(0x7f9f6806a370 msgr2=0x7f9f680a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 -- 172.21.15.12:0/803121029 shutdown_connections 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.413+0000 7f9f6d33f700 1 -- 172.21.15.12:0/803121029 wait complete. 2023-09-29T23:13:39.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f1c28f2f700 1 -- 172.21.15.12:0/1896347256 wait complete. 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f3df7ff9700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f3df7ff9700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f3df7ff9700 10 monclient: monmap: 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.415060+0000 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.415060+0000 2023-09-29T23:13:39.453 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.454 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f9f6d33f700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f9f6d33f700 10 monclient: monmap: 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.455 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.456 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f3df7ff9700 1 Processor -- start 2023-09-29T23:13:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.414+0000 7f9f6d33f700 1 Processor -- start 2023-09-29T23:13:39.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 -- start start 2023-09-29T23:13:39.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient: init 2023-09-29T23:13:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 -- start start 2023-09-29T23:13:39.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient: init 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7fbba9a95700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7fbba9a95700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7fbba9a95700 10 monclient: monmap: 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.416133+0000 2023-09-29T23:13:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.416133+0000 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.463 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient: _add_conns ranks=[1,8,9,5,3,4,7,10,2,6,0] 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient: _add_conns ranks=[0,6,2,1,7,5,3,8,4] 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f6809adf0 0x7f9f681342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-a con 0x7f9f6809adf0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 0x7f9f6811fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-d con 0x7f9f6809d5a0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3df009b410 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3df009a9a0 0x7f3df009cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-e con 0x7f3df009a9a0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f3df009b7e0 0x7f3df00a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f3df009b7e0 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0068ed0 con 0x7f3df009b410 2023-09-29T23:13:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df009c720 con 0x7f3df009b7e0 2023-09-29T23:13:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00a26d0 con 0x7f3df009a9a0 2023-09-29T23:13:39.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f3df7ff9700 10 monclient(hunting): authenticate will time out at 1128.775831s 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f681347e0 0x7f9f681201c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): picked mon.noname-b con 0x7f9f681347e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f681225f0 con 0x7f9f6809adf0 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f68121ea0 con 0x7f9f681347e0 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6809b250 con 0x7f9f6809d5a0 2023-09-29T23:13:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.415+0000 7f9f6d33f700 10 monclient(hunting): authenticate will time out at 1128.775807s 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 0x7f9f6811fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 0x7f9f6811fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56258/0 (socket says 172.21.15.12:56258) 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 -- 172.21.15.12:0/693848549 learned_addr learned my addr 172.21.15.12:0/693848549 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f681347e0 0x7f9f681201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): get_auth_request con 0x7f9f6809d5a0 auth_method 0 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): get_auth_request con 0x7f9f681347e0 auth_method 0 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f6809adf0 0x7f9f681342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f3df5d95700 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3df009b410 0x7f3df00a10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:36570/0 (socket says 172.21.15.12:36570) 2023-09-29T23:13:39.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f3df5d95700 1 -- 172.21.15.12:0/679205302 learned_addr learned my addr 172.21.15.12:0/679205302 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:39.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): get_auth_request con 0x7f9f6809adf0 auth_method 0 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 -- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f681347e0 msgr2=0x7f9f681201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9f681347e0 0x7f9f681201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 -- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f6809adf0 msgr2=0x7f9f681342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9f6809adf0 0x7f9f681342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 monclient: found mon.noname-d 2023-09-29T23:13:39.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 -- 172.21.15.12:0/693848549 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f50009420 con 0x7f9f6809d5a0 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f667fc700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 0x7f9f6811fc80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800ece0 tx=0x7f9f580091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f5800ccd0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.416+0000 7f9f5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:39.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 --2- 172.21.15.12:0/679205302 >> v2:172.21.15.12:3300/0 conn(0x7f3df009b7e0 0x7f3df00a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): get_auth_request con 0x7f3df009b7e0 auth_method 0 2023-09-29T23:13:39.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: dump: 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.476 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5800ce70 con 0x7f9f6809d5a0 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 5 monclient: authenticate success, global_id 4308 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 10 monclient: _renew_subs 2023-09-29T23:13:39.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6804c960 con 0x7f9f6809d5a0 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f58013470 con 0x7f9f6809d5a0 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f657fa700 4 set_mon_vals no callback set 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f5effd700 10 monclient: dump: 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.478 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 10 monclient: _renew_subs 2023-09-29T23:13:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f681236c0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 -- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3df009a9a0 msgr2=0x7f3df009cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 --2- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3df009a9a0 0x7f3df009cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 -- 172.21.15.12:0/679205302 >> v1:172.21.15.153:6789/0 conn(0x7f3df009b410 legacy=0x7f3df00a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 -- 172.21.15.12:0/679205302 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df004c960 con 0x7f3df009b7e0 2023-09-29T23:13:39.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.417+0000 7f3df6596700 1 --2- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df00a17d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009ec0 tx=0x7f3de8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7ac5c3ceb23e96 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f6d33f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 Processor -- start 2023-09-29T23:13:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f6809e9d0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 -- start start 2023-09-29T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient: init 2023-09-29T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9f5800c850 con 0x7f9f6809d5a0 2023-09-29T23:13:39.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f5effd700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6812/83257 conn(0x7f9f48036850 0x7f9f48038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 1 -- 172.21.15.12:0/679205302 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de800e070 con 0x7f3df009b7e0 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: dump: 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 1 -- 172.21.15.12:0/679205302 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de800bbb0 con 0x7f3df009b7e0 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df7ff9700 5 monclient: authenticate success, global_id 4338 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df7ff9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 1 -- 172.21.15.12:0/679205302 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de8012d30 con 0x7f3df009b7e0 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df4d93700 10 monclient: dump: 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df7ff9700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f3df7ff9700 4 set_mon_vals no callback set 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9f580476c0 con 0x7f9f6809d5a0 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f5effd700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6803/83794 conn(0x7f9f4803c2f0 0x7f9f4803e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 --> v2:172.21.15.153:6803/83794 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4803ed70 con 0x7f9f4803c2f0 2023-09-29T23:13:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6812/83257 conn(0x7f9f48036850 0x7f9f48038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7f9f66ffd700 10 monclient: get_auth_request con 0x7f9f48036850 auth_method 0 2023-09-29T23:13:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient: _add_conns ranks=[4,2,10,9,8,0,1,5,7,3,6] 2023-09-29T23:13:39.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-c con 0x7fbba412f660 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 0x7fbba406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-b con 0x7fbba406eb60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbba409d5a0 0x7fbba409da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-f con 0x7fbba409d5a0 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba406bcf0 con 0x7fbba409d5a0 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba406d090 con 0x7fbba406eb60 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.418+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba9a95700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbba412fa30 con 0x7fbba412f660 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba9a95700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba9a95700 10 monclient(hunting): authenticate will time out at 1128.779019s 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 0x7fbba406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37830/0 (socket says 172.21.15.12:37830) 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 -- 172.21.15.12:0/1129115935 learned_addr learned my addr 172.21.15.12:0/1129115935 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): get_auth_request con 0x7fbba412f660 auth_method 0 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): get_auth_request con 0x7fbba406eb60 auth_method 0 2023-09-29T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f9f66ffd700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6812/83257 conn(0x7f9f48036850 0x7f9f48038d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009910 tx=0x7f9f50017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 10 monclient: shutdown 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 -- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 msgr2=0x7f3df00a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 --2- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df00a17d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009ec0 tx=0x7f3de8009780 comp rx=0 tx=0).stop 2023-09-29T23:13:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-09-29T23:13:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 -- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 msgr2=0x7fbba406ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 0x7fbba406ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 -- 172.21.15.12:0/1129115935 >> v2:172.21.15.153:3300/0 conn(0x7fbba409d5a0 msgr2=0x7fbba409da10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 --2- 172.21.15.12:0/1129115935 >> v2:172.21.15.153:3300/0 conn(0x7fbba409d5a0 0x7fbba409da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba37fe700 1 -- 172.21.15.12:0/1129115935 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4068ed0 con 0x7fbba412f660 2023-09-29T23:13:39.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7fbba2ffd700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 0x7fbba406ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 -- 172.21.15.12:0/679205302 shutdown_connections 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 --2- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df00a17d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 --2- 172.21.15.12:0/679205302 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3df009a9a0 0x7f3df009cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 -- 172.21.15.12:0/679205302 >> 172.21.15.12:0/679205302 conn(0x7f3df0063d70 msgr2=0x7f3df009fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 -- 172.21.15.12:0/679205302 shutdown_connections 2023-09-29T23:13:39.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.419+0000 7f3df7ff9700 1 -- 172.21.15.12:0/679205302 wait complete. 2023-09-29T23:13:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba37fe700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb940099b0 tx=0x7fbb94009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=65fa7e2ca8faef36 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 1 -- 172.21.15.12:0/1129115935 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb94009ea0 con 0x7fbba412f660 2023-09-29T23:13:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: dump: 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.498 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba9a95700 5 monclient: authenticate success, global_id 4315 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 1 -- 172.21.15.12:0/1129115935 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9400bce0 con 0x7fbba412f660 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba9a95700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba9a95700 4 set_mon_vals no callback set 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 1 -- 172.21.15.12:0/1129115935 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb9400be50 con 0x7fbba412f660 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7fbba27fc700 10 monclient: dump: 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.500 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f9f677fe700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6803/83794 conn(0x7f9f4803c2f0 0x7f9f4803e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7f3df7ff9700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.420+0000 7f3df7ff9700 10 monclient: monmap: 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.502 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.503 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 Processor -- start 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 -- start start 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient: init 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7fbba9a95700 10 monclient: shutdown 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7fbba9a95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7fbba9a95700 1 -- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 msgr2=0x7fbba412fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7fbba9a95700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb940099b0 tx=0x7fbb94009250 comp rx=0 tx=0).stop 2023-09-29T23:13:39.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient: _add_conns ranks=[1,7,0,6,8,3,4,5,2] 2023-09-29T23:13:39.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df009a9a0 0x7f3df0134900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-f con 0x7f3df009a9a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 0x7f3df0134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-i con 0x7f3df009b410 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df0135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f3df009b7e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0099a10 con 0x7f3df009b7e0 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df006f580 con 0x7f3df009a9a0 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0137f80 con 0x7f3df009b410 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df7ff9700 10 monclient(hunting): authenticate will time out at 1128.781733s 2023-09-29T23:13:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f9f677fe700 10 monclient: get_auth_request con 0x7f9f4803c2f0 auth_method 0 2023-09-29T23:13:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f9f677fe700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6803/83794 conn(0x7f9f4803c2f0 0x7f9f4803e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 0x7f3df0134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5d95700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df009a9a0 0x7f3df0134900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df6596700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df0135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 0x7f3df0134e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:56220/0 (socket says 172.21.15.12:56220) 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 1 -- 172.21.15.12:0/545110692 learned_addr learned my addr 172.21.15.12:0/545110692 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5d95700 10 monclient(hunting): get_auth_request con 0x7f3df009a9a0 auth_method 0 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5d95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5d95700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5d95700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 10 monclient(hunting): get_auth_request con 0x7f3df009b410 auth_method 0 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df5594700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df6596700 10 monclient(hunting): get_auth_request con 0x7f3df009b7e0 auth_method 0 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.421+0000 7f3df6596700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5d95700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5d95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5d95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5594700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5594700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5594700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 -- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 msgr2=0x7f3df0134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 0x7f3df0134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 -- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df009a9a0 msgr2=0x7f3df0134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df009a9a0 0x7f3df0134900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 -- 172.21.15.12:0/545110692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8009a20 con 0x7f3df009b7e0 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5594700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3df009b410 0x7f3df0134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df5d95700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3df009a9a0 0x7f3df0134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f3df6596700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df0135380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3df00a0d80 tx=0x7f3de8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.422+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== osd.4 v2:172.21.15.153:6803/83794 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f9f4803ed70 con 0x7f9f4803c2f0 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.423+0000 7fbba9a95700 1 -- 172.21.15.12:0/1129115935 shutdown_connections 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.423+0000 7fbba9a95700 1 --2- 172.21.15.12:0/1129115935 >> v2:172.21.15.153:3300/0 conn(0x7fbba409d5a0 0x7fbba409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.423+0000 7fbba9a95700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412fca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.423+0000 7fbba9a95700 1 --2- 172.21.15.12:0/1129115935 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbba406eb60 0x7fbba406ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.423+0000 7fbba9a95700 1 -- 172.21.15.12:0/1129115935 >> 172.21.15.12:0/1129115935 conn(0x7fbba406a370 msgr2=0x7fbba40a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de801c410 con 0x7f3df009b7e0 2023-09-29T23:13:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: dump: 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de8021070 con 0x7f3df009b7e0 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 5 monclient: authenticate success, global_id 4347 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de801ca50 con 0x7f3df009b7e0 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de7fff700 4 set_mon_vals no callback set 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3de5ffb700 10 monclient: dump: 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.516 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.517 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.518 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 10 monclient: _renew_subs 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 1 -- 172.21.15.12:0/545110692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df0136450 con 0x7f3df009b7e0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 10 monclient: _renew_subs 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.424+0000 7f3df7ff9700 1 -- 172.21.15.12:0/545110692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df0137090 con 0x7f3df009b7e0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.425+0000 7f3dd37fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.425+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3dcc000f10 con 0x7f3df009b7e0 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3de8023030 con 0x7f3df009b7e0 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7fbba9a95700 1 -- 172.21.15.12:0/1129115935 shutdown_connections 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7fbba9a95700 1 -- 172.21.15.12:0/1129115935 wait complete. 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7fbba9a95700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7fbba9a95700 10 monclient: monmap: 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.520 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.521 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f58013b10 con 0x7f9f6809d5a0 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f9f5effd700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f401d320700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f401d320700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.426+0000 7f401d320700 10 monclient: monmap: 2023-09-29T23:13:39.522 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.427417+0000 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.427417+0000 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.523 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.153:6812/83257 conn(0x7f3dd4036780 0x7f3dd4038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3de80496e0 con 0x7f3df009b7e0 2023-09-29T23:13:39.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f401d320700 1 Processor -- start 2023-09-29T23:13:39.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.12:6800/83762 conn(0x7f3dd403c2a0 0x7f3dd403e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 --> v2:172.21.15.12:6800/83762 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dd403ed20 con 0x7f3dd403c2a0 2023-09-29T23:13:39.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3de8049a80 con 0x7f3df009b7e0 2023-09-29T23:13:39.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3de5ffb700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7fbba9a95700 1 Processor -- start 2023-09-29T23:13:39.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3df5d95700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.153:6812/83257 conn(0x7f3dd4036780 0x7f3dd4038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f3df5d95700 10 monclient: get_auth_request con 0x7f3dd4036780 auth_method 0 2023-09-29T23:13:39.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7fbba9a95700 1 -- start start 2023-09-29T23:13:39.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7fbba9a95700 10 monclient: init 2023-09-29T23:13:39.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f3df5d95700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.153:6812/83257 conn(0x7f3dd4036780 0x7f3dd4038c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0017e0 tx=0x7f3dec007eb0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7fbba9a95700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7fbba9a95700 10 monclient: _add_conns ranks=[5,3,4,7,1,2,6,8,0] 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbba406eb60 0x7fbba4146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-h con 0x7fbba406eb60 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba409d5a0 0x7fbba4147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-g con 0x7fbba409d5a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): picked mon.noname-c con 0x7fbba412f660 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba406c170 con 0x7fbba409d5a0 2023-09-29T23:13:39.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4123c60 con 0x7fbba412f660 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4124190 con 0x7fbba406eb60 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba9a95700 10 monclient(hunting): authenticate will time out at 1128.788075s 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37838/0 (socket says 172.21.15.12:37838) 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 -- 172.21.15.12:0/2326142670 learned_addr learned my addr 172.21.15.12:0/2326142670 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): get_auth_request con 0x7fbba412f660 auth_method 0 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba409d5a0 0x7fbba4147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 10 monclient(hunting): get_auth_request con 0x7fbba409d5a0 auth_method 0 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f401d320700 1 -- start start 2023-09-29T23:13:39.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f401d320700 10 monclient: init 2023-09-29T23:13:39.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f401d320700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.427+0000 7f401d320700 10 monclient: _add_conns ranks=[9,4,1,7,0,8,5,6,10,2,3] 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f401812f660 0x7f401812fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): picked mon.noname-a con 0x7f401812f660 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f401806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): picked mon.noname-c con 0x7f401806eb60 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f401809d5a0 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4018068ed0 con 0x7f401809d5a0 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401806bcf0 con 0x7f401812f660 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f401806d090 con 0x7f401806eb60 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f401d320700 10 monclient(hunting): authenticate will time out at 1128.788106s 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f401812f660 0x7f401812fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f401806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f401812f660 0x7f401812fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52896/0 (socket says 172.21.15.12:52896) 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 -- 172.21.15.12:0/3303045845 learned_addr learned my addr 172.21.15.12:0/3303045845 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): get_auth_request con 0x7f401806eb60 auth_method 0 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): get_auth_request con 0x7f401812f660 auth_method 0 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 -- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 msgr2=0x7f401806ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 --2- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f401806ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 -- 172.21.15.12:0/3303045845 >> v1:172.21.15.153:6789/0 conn(0x7f401809d5a0 legacy=0x7f401809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4015ffb700 1 -- 172.21.15.12:0/3303045845 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553616027 0 0) 0x7f4018068ed0 con 0x7f401809d5a0 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 -- 172.21.15.12:0/3303045845 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401804c960 con 0x7f401812f660 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4015ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2023-09-29T23:13:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 -- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbba406eb60 msgr2=0x7fbba4146d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbba406eb60 0x7fbba4146d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 -- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba409d5a0 msgr2=0x7fbba4147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba409d5a0 0x7fbba4147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 -- 172.21.15.12:0/2326142670 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98009420 con 0x7fbba412f660 2023-09-29T23:13:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba2ffd700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba409d5a0 0x7fbba4147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7f4016ffd700 1 --2- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f401812f660 0x7f401812fca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f400800ddd0 tx=0x7f400800d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6467ae40155847 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.428+0000 7fbba3fff700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412cf10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00bbb0 tx=0x7fbb9c009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 1 -- 172.21.15.12:0/3303045845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4008012070 con 0x7f401812f660 2023-09-29T23:13:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb9c0092e0 con 0x7fbba412f660 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: dump: 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 10 monclient: dump: 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.540 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 1 -- 172.21.15.12:0/3303045845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400800d2e0 con 0x7f401812f660 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 5 monclient: authenticate success, global_id 4333 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9c009980 con 0x7fbba412f660 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 10 monclient: _renew_subs 2023-09-29T23:13:39.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb940094f0 con 0x7fbba412f660 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 1 -- 172.21.15.12:0/3303045845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f400800fdd0 con 0x7f401812f660 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f4015ffb700 10 monclient: dump: 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.544 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 10 monclient: _renew_subs 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba404c960 con 0x7fbba412f660 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f401d320700 5 monclient: authenticate success, global_id 4356 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f401d320700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f401d320700 4 set_mon_vals no callback set 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f401d320700 10 monclient: shutdown 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.429+0000 7f401d320700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 -- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f401812f660 msgr2=0x7f401812fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 --2- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f401812f660 0x7f401812fca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f400800ddd0 tx=0x7f400800d6f0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbb9c00cc20 con 0x7fbba412f660 2023-09-29T23:13:39.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 10 monclient: dump: 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.547 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 -- 172.21.15.12:0/3303045845 shutdown_connections 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 --2- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f401812f660 0x7f401812fca0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 --2- 172.21.15.12:0/3303045845 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f401806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 -- 172.21.15.12:0/3303045845 >> 172.21.15.12:0/3303045845 conn(0x7f401806a370 msgr2=0x7f40180a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 -- 172.21.15.12:0/3303045845 shutdown_connections 2023-09-29T23:13:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba9a95700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbba4070bc0 con 0x7fbba412f660 2023-09-29T23:13:39.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7f401d320700 1 -- 172.21.15.12:0/3303045845 wait complete. 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbb9c012070 con 0x7fbba412f660 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba17fa700 4 set_mon_vals no callback set 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6812/83257 conn(0x7fbb90036780 0x7fbb90038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbb9c015490 con 0x7fbba412f660 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6800/83793 conn(0x7fbb9003c220 0x7fbb9003e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 --> v2:172.21.15.153:6800/83793 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9003eca0 con 0x7fbb9003c220 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb9c049590 con 0x7fbba412f660 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbb8b7fe700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba37fe700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6812/83257 conn(0x7fbb90036780 0x7fbb90038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba37fe700 10 monclient: get_auth_request con 0x7fbb90036780 auth_method 0 2023-09-29T23:13:39.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.430+0000 7fbba2ffd700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6800/83793 conn(0x7fbb9003c220 0x7fbb9003e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7fbba2ffd700 10 monclient: get_auth_request con 0x7fbb9003c220 auth_method 0 2023-09-29T23:13:39.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7fbba37fe700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6812/83257 conn(0x7fbb90036780 0x7fbb90038c30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbb940099b0 tx=0x7fbb94017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7fbba2ffd700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6800/83793 conn(0x7fbb9003c220 0x7fbb9003e630 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7f401d320700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7f401d320700 10 monclient: monmap: 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7f401d320700 1 Processor -- start 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7f401d320700 1 -- start start 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.431+0000 7f401d320700 10 monclient: init 2023-09-29T23:13:39.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient: _add_conns ranks=[4,2,8,1,7,0,5,3,6] 2023-09-29T23:13:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f4018146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): picked mon.noname-c con 0x7f401806eb60 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== osd.3 v2:172.21.15.153:6800/83793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7fbb9003eca0 con 0x7fbb9003c220 2023-09-29T23:13:39.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f401809d5a0 0x7f4018146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): picked mon.noname-b con 0x7f401809d5a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f401812f660 0x7f401809b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): picked mon.noname-e con 0x7f401812f660 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4018124a30 con 0x7f401809d5a0 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4018123b80 con 0x7f401806eb60 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f401806c170 con 0x7f401812f660 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f401d320700 10 monclient(hunting): authenticate will time out at 1128.792279s 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f4018146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f401812f660 0x7f401809b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f4018146a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37862/0 (socket says 172.21.15.12:37862) 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 -- 172.21.15.12:0/3488656415 learned_addr learned my addr 172.21.15.12:0/3488656415 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): get_auth_request con 0x7f401806eb60 auth_method 0 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): get_auth_request con 0x7f401812f660 auth_method 0 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f401809d5a0 0x7f4018146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): get_auth_request con 0x7f401809d5a0 auth_method 0 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-09-29T23:13:39.560 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 -- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f401812f660 msgr2=0x7f401809b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f401812f660 0x7f401809b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 -- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f401809d5a0 msgr2=0x7f4018146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f401809d5a0 0x7f4018146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f4016ffd700 1 -- 172.21.15.12:0/3488656415 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400800d990 con 0x7f401806eb60 2023-09-29T23:13:39.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:13:39.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40177fe700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f401812f660 0x7f401809b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.432+0000 7f40167fc700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f401809d5a0 0x7f4018146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3df5594700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.12:6800/83762 conn(0x7f3dd403c2a0 0x7f3dd403e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f4016ffd700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f4018146a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f401806be30 tx=0x7f400801c000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3df5594700 10 monclient: get_auth_request con 0x7f3dd403c2a0 auth_method 0 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f400800b070 con 0x7f401806eb60 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: dump: 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.564 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f401d320700 5 monclient: authenticate success, global_id 4342 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4008016c80 con 0x7f401806eb60 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4008016df0 con 0x7f401806eb60 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f40157fa700 4 set_mon_vals no callback set 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3ffeffd700 10 monclient: dump: 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.566 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.568 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.433+0000 7f3df5594700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.12:6800/83762 conn(0x7f3dd403c2a0 0x7f3dd403e6b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== osd.1 v2:172.21.15.12:6800/83762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f3dd403ed20 con 0x7f3dd403c2a0 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 10 monclient: _renew_subs 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40181240b0 con 0x7f401806eb60 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 10 monclient: _renew_subs 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.434+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018123810 con 0x7f401806eb60 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4008016800 con 0x7f401806eb60 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f401d320700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4018070bc0 con 0x7f401806eb60 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6812/83257 conn(0x7f4004036850 0x7f4004038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4008047e30 con 0x7f401806eb60 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f40167fc700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6812/83257 conn(0x7f4004036850 0x7f4004038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6808/83795 conn(0x7f400403c2f0 0x7f400403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 --> v2:172.21.15.153:6808/83795 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f400403ed70 con 0x7f400403c2f0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f40167fc700 10 monclient: get_auth_request con 0x7f4004036850 auth_method 0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4008028460 con 0x7f401806eb60 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f3ffeffd700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f40177fe700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6808/83795 conn(0x7f400403c2f0 0x7f400403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.435+0000 7f40177fe700 10 monclient: get_auth_request con 0x7f400403c2f0 auth_method 0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.436+0000 7f40167fc700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6812/83257 conn(0x7f4004036850 0x7f4004038d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f400c00bfd0 tx=0x7f400c00f040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.436+0000 7f40177fe700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6808/83795 conn(0x7f400403c2f0 0x7f400403e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.436+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== osd.5 v2:172.21.15.153:6808/83795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f400403ed70 con 0x7f400403c2f0 2023-09-29T23:13:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.444+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 --> v2:172.21.15.153:6803/83794 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f681238b0 con 0x7f9f4803c2f0 2023-09-29T23:13:39.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f5effd700 1 -- 172.21.15.12:0/693848549 <== osd.4 v2:172.21.15.153:6803/83794 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9f681238b0 con 0x7f9f4803c2f0 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6803/83794 conn(0x7f9f4803c2f0 msgr2=0x7f9f4803e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6803/83794 conn(0x7f9f4803c2f0 0x7f9f4803e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6812/83257 conn(0x7f9f48036850 msgr2=0x7f9f48038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/693848549 >> v2:172.21.15.153:6812/83257 conn(0x7f9f48036850 0x7f9f48038d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009910 tx=0x7f9f50017000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 10 monclient: shutdown 2023-09-29T23:13:39.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 msgr2=0x7f9f6811fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 --2- 172.21.15.12:0/693848549 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9f6809d5a0 0x7f9f6811fc80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800ece0 tx=0x7f9f580091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f677fe700 1 -- 172.21.15.12:0/693848549 reap_dead start 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 shutdown_connections 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 >> 172.21.15.12:0/693848549 conn(0x7f9f6806a370 msgr2=0x7f9f6806d340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 shutdown_connections 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.446+0000 7f9f6d33f700 1 -- 172.21.15.12:0/693848549 wait complete. 2023-09-29T23:13:39.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.448+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 --> v2:172.21.15.153:6800/83793 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbba41229b0 con 0x7fbb9003c220 2023-09-29T23:13:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 --> v2:172.21.15.12:6800/83762 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3dcc002bc0 con 0x7f3dd403c2a0 2023-09-29T23:13:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbb8b7fe700 1 -- 172.21.15.12:0/2326142670 <== osd.3 v2:172.21.15.153:6800/83793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbba41229b0 con 0x7fbb9003c220 2023-09-29T23:13:39.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3de5ffb700 1 -- 172.21.15.12:0/545110692 <== osd.1 v2:172.21.15.12:6800/83762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3dcc002bc0 con 0x7f3dd403c2a0 2023-09-29T23:13:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6800/83793 conn(0x7fbb9003c220 msgr2=0x7fbb9003e630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6800/83793 conn(0x7fbb9003c220 0x7fbb9003e630 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6812/83257 conn(0x7fbb90036780 msgr2=0x7fbb90038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 --2- 172.21.15.12:0/2326142670 >> v2:172.21.15.153:6812/83257 conn(0x7fbb90036780 0x7fbb90038c30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbb940099b0 tx=0x7fbb94017000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 10 monclient: shutdown 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 msgr2=0x7fbba412cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 --2- 172.21.15.12:0/2326142670 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbba412f660 0x7fbba412cf10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00bbb0 tx=0x7fbb9c009840 comp rx=0 tx=0).stop 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba3fff700 1 -- 172.21.15.12:0/2326142670 reap_dead start 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 >> v2:172.21.15.12:6800/83762 conn(0x7f3dd403c2a0 msgr2=0x7f3dd403e6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.12:6800/83762 conn(0x7f3dd403c2a0 0x7f3dd403e6b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 shutdown_connections 2023-09-29T23:13:39.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 >> 172.21.15.12:0/2326142670 conn(0x7fbba406a370 msgr2=0x7fbba4128610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 shutdown_connections 2023-09-29T23:13:39.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7fbba9a95700 1 -- 172.21.15.12:0/2326142670 wait complete. 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 >> v2:172.21.15.153:6812/83257 conn(0x7f3dd4036780 msgr2=0x7f3dd4038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 --2- 172.21.15.12:0/545110692 >> v2:172.21.15.153:6812/83257 conn(0x7f3dd4036780 0x7f3dd4038c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0017e0 tx=0x7f3dec007eb0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 10 monclient: shutdown 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 msgr2=0x7f3df0135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3dd37fe700 1 --2- 172.21.15.12:0/545110692 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3df009b7e0 0x7f3df0135380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3df00a0d80 tx=0x7f3de8019000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.449+0000 7f3df6596700 1 -- 172.21.15.12:0/545110692 reap_dead start 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.451+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 shutdown_connections 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.451+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 >> 172.21.15.12:0/545110692 conn(0x7f3df0063d70 msgr2=0x7f3df00989f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.451+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 shutdown_connections 2023-09-29T23:13:39.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.451+0000 7f3dd37fe700 1 -- 172.21.15.12:0/545110692 wait complete. 2023-09-29T23:13:39.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.455+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 --> v2:172.21.15.153:6808/83795 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40181228d0 con 0x7f400403c2f0 2023-09-29T23:13:39.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f3ffeffd700 1 -- 172.21.15.12:0/3488656415 <== osd.5 v2:172.21.15.153:6808/83795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40181228d0 con 0x7f400403c2f0 2023-09-29T23:13:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6808/83795 conn(0x7f400403c2f0 msgr2=0x7f400403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6808/83795 conn(0x7f400403c2f0 0x7f400403e700 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6812/83257 conn(0x7f4004036850 msgr2=0x7f4004038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 --2- 172.21.15.12:0/3488656415 >> v2:172.21.15.153:6812/83257 conn(0x7f4004036850 0x7f4004038d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f400c00bfd0 tx=0x7f400c00f040 comp rx=0 tx=0).stop 2023-09-29T23:13:39.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 10 monclient: shutdown 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 msgr2=0x7f4018146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 --2- 172.21.15.12:0/3488656415 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f401806eb60 0x7f4018146a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f401806be30 tx=0x7f400801c000 comp rx=0 tx=0).stop 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f40177fe700 1 -- 172.21.15.12:0/3488656415 reap_dead start 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 shutdown_connections 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 >> 172.21.15.12:0/3488656415 conn(0x7f401806a370 msgr2=0x7f401812ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 shutdown_connections 2023-09-29T23:13:39.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.456+0000 7f401d320700 1 -- 172.21.15.12:0/3488656415 wait complete. 2023-09-29T23:13:39.583 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-29T23:13:39.584 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.584 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-29T23:13:39.585 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-29T23:13:39.586 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:39.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:13:39.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 10 monclient: monmap: 2023-09-29T23:13:39.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.735996+0000 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.735996+0000 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.736 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.737 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.737 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.737 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.737 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.737 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 1 Processor -- start 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 1 -- start start 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.735+0000 7fb88c6fc700 10 monclient: init 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient: _add_conns ranks=[5,0,8,10,4,3,2,1,6,7,9] 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb88409d5a0 0x7fb88409d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-h con 0x7fb88409d5a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb88409cba0 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-e con 0x7fb88409adf0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:39.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884068ed0 con 0x7fb88409cba0 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb88409bed0 con 0x7fb88409d5a0 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb88409bbf0 con 0x7fb88409adf0 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88c6fc700 10 monclient(hunting): authenticate will time out at 1129.096618s 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55338/0 (socket says 172.21.15.12:55338) 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 1 -- 172.21.15.12:0/4176267829 learned_addr learned my addr 172.21.15.12:0/4176267829 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 10 monclient(hunting): get_auth_request con 0x7fb88409adf0 auth_method 0 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.736+0000 7fb88ac99700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88a498700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb88409d5a0 0x7fb88409d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 -- 172.21.15.12:0/4176267829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb88409d5a0 msgr2=0x7fb88409d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb88409d5a0 0x7fb88409d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 -- 172.21.15.12:0/4176267829 >> v1:172.21.15.12:6789/0 conn(0x7fb88409cba0 legacy=0x7fb8841333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 monclient: found mon.noname-e 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 -- 172.21.15.12:0/4176267829 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88404c960 con 0x7fb88409adf0 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7f2779f88700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7f2779f88700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7f2779f88700 10 monclient: monmap: 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.738136+0000 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.738136+0000 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.743 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.744 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.744 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.744 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb88ac99700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009f50 tx=0x7fb87c009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=9d63f677a60713ad server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 1 -- 172.21.15.12:0/4176267829 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb87c00e070 con 0x7fb88409adf0 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 10 monclient: dump: 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.746 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.737+0000 7fb889496700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 1 -- 172.21.15.12:0/4176267829 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c007400 con 0x7fb88409adf0 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 1 -- 172.21.15.12:0/4176267829 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb87c0092e0 con 0x7fb88409adf0 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:13:39.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb889496700 10 monclient: dump: 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.748 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 5 monclient: authenticate success, global_id 4247 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 4 set_mon_vals no callback set 2023-09-29T23:13:39.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 Processor -- start 2023-09-29T23:13:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 -- start start 2023-09-29T23:13:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient: init 2023-09-29T23:13:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 10 monclient: shutdown 2023-09-29T23:13:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 1 -- 172.21.15.12:0/4176267829 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 msgr2=0x7fb88409deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009f50 tx=0x7fb87c009840 comp rx=0 tx=0).stop 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 -- 172.21.15.12:0/4176267829 shutdown_connections 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb88409adf0 0x7fb88409deb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/4176267829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb88409d5a0 0x7fb88409d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient: _add_conns ranks=[0,1,7,4,3,6,8,9,5,10,2] 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2774099660 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:39.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f277409a870 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f277409a140 0x7f277409d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-i con 0x7f277409a140 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2774068ed0 con 0x7f2774099660 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277404c960 con 0x7f277409a870 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f277409bc50 con 0x7f277409a140 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.738+0000 7f2779f88700 10 monclient(hunting): authenticate will time out at 1129.098980s 2023-09-29T23:13:39.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 -- 172.21.15.12:0/4176267829 >> 172.21.15.12:0/4176267829 conn(0x7fb88406a370 msgr2=0x7fb8840a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 -- 172.21.15.12:0/4176267829 shutdown_connections 2023-09-29T23:13:39.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 1 -- 172.21.15.12:0/4176267829 wait complete. 2023-09-29T23:13:39.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27737fe700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f2774099660 0x7f2774096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37960/0 (socket says 172.21.15.12:37960) 2023-09-29T23:13:39.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27737fe700 1 -- 172.21.15.12:0/3688257681 learned_addr learned my addr 172.21.15.12:0/3688257681 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:39.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f277409a140 0x7f277409d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): get_auth_request con 0x7f277409a140 auth_method 0 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622399935 0 0) 0x7f2774068ed0 con 0x7f2774099660 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): my global_id is 4365 2023-09-29T23:13:39.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f275c003570 con 0x7f2774099660 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883735224 0 0) 0x7f277404c960 con 0x7f277409a870 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 10 monclient(hunting): my global_id is 4285 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774068ed0 con 0x7f277409a870 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2968541961 0 0) 0x7f275c003570 con 0x7f2774099660 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f277409a140 msgr2=0x7f277409d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f277409a140 0x7f277409d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 >> v1:172.21.15.153:6789/0 conn(0x7f277409a870 legacy=0x7f2774099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740966a0 con 0x7f2774099660 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f27640018f0 con 0x7f2774099660 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: dump: 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.758 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.759 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 >> v1:172.21.15.12:6789/0 conn(0x7f2774099660 legacy=0x7f2774096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient: _add_conns ranks=[4,5,7,3,6,1,0,8,2] 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f275c004810 0x7f275c004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): picked mon.c con 0x7f275c004810 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f275c005120 0x7f275c009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): picked mon.h con 0x7f275c005120 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f275c009f90 0x7f275c00c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): picked mon.i con 0x7f275c009f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f277409bc50 con 0x7f275c004810 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f275c00c960 con 0x7f275c005120 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f275c00cad0 con 0x7f275c009f90 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27727fc700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:13:39.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f275c004810 0x7f275c004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): get_auth_request con 0x7f275c004810 auth_method 0 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.739+0000 7fb88c6fc700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7fb88c6fc700 10 monclient: monmap: 2023-09-29T23:13:39.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7fb88c6fc700 1 Processor -- start 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f275c005120 0x7f275c009a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): get_auth_request con 0x7f275c005120 auth_method 0 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient(hunting): handle_auth_done global_id 4365 payload 293 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 -- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f275c009f90 msgr2=0x7f275c00c370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f275c009f90 0x7f275c00c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 -- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f275c005120 msgr2=0x7f275c009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f275c005120 0x7f275c009a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 monclient: found mon.c 2023-09-29T23:13:39.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 -- 172.21.15.12:0/3688257681 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740966a0 con 0x7f275c004810 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2779f88700 5 monclient: authenticate success, global_id 4365 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f2779f88700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7f27737fe700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f275c004810 0x7f275c004be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2764001ed0 tx=0x7f2764001670 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1c20628f33cc4a7c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2764007980 con 0x7f275c004810 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: dump: 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.769 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276400ce60 con 0x7f275c004810 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f2779f88700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 1 -- 172.21.15.12:0/3688257681 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2764005da0 con 0x7f275c004810 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f27727fc700 10 monclient: dump: 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.771 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f2779f88700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7f2779f88700 4 set_mon_vals no callback set 2023-09-29T23:13:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7fb88c6fc700 1 -- start start 2023-09-29T23:13:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.740+0000 7fb88c6fc700 10 monclient: init 2023-09-29T23:13:39.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient: _add_conns ranks=[0,2,1,7,6,5,8,3,4] 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-a con 0x7fb88409adf0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb88409d5a0 0x7fb884142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-b con 0x7fb88409d5a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb884143a70 0x7fb8841465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): picked mon.noname-f con 0x7fb884143a70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88409a720 con 0x7fb88409adf0 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88409b250 con 0x7fb884143a70 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88409a440 con 0x7fb88409d5a0 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.741+0000 7fb88c6fc700 10 monclient(hunting): authenticate will time out at 1129.101534s 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 10 monclient: shutdown 2023-09-29T23:13:39.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 -- 172.21.15.12:0/3688257681 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f275c004810 msgr2=0x7f275c004be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 --2- 172.21.15.12:0/3688257681 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f275c004810 0x7f275c004be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2764001ed0 tx=0x7f2764001670 comp rx=0 tx=0).stop 2023-09-29T23:13:39.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2773fff700 1 -- 172.21.15.12:0/3688257681 reap_dead start 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52908/0 (socket says 172.21.15.12:52908) 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 -- 172.21.15.12:0/365538434 learned_addr learned my addr 172.21.15.12:0/365538434 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): get_auth_request con 0x7fb88409adf0 auth_method 0 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 -- 172.21.15.12:0/3688257681 shutdown_connections 2023-09-29T23:13:39.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 -- 172.21.15.12:0/3688257681 >> 172.21.15.12:0/3688257681 conn(0x7f277409ee30 msgr2=0x7f277409f230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 -- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb88409d5a0 msgr2=0x7fb884142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb88409d5a0 0x7fb884142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 -- 172.21.15.12:0/365538434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb884143a70 msgr2=0x7fb8841465d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb884143a70 0x7fb8841465d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7fb88a498700 1 -- 172.21.15.12:0/365538434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009ae0 con 0x7fb88409adf0 2023-09-29T23:13:39.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 -- 172.21.15.12:0/3688257681 shutdown_connections 2023-09-29T23:13:39.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.742+0000 7f2779f88700 1 -- 172.21.15.12:0/3688257681 wait complete. 2023-09-29T23:13:39.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88a498700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c800 tx=0x7fb880008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7f2779f88700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7f2779f88700 10 monclient: monmap: 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.782 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb880011a80 con 0x7fb88409adf0 2023-09-29T23:13:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: dump: 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.784 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb880011c20 con 0x7fb88409adf0 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb880011d90 con 0x7fb88409adf0 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.785 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb87a7fc700 10 monclient: dump: 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.786 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7f2779f88700 1 Processor -- start 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb888c95700 4 set_mon_vals no callback set 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 5 monclient: authenticate success, global_id 4374 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 10 monclient: _renew_subs 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88409a160 con 0x7fb88409adf0 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 10 monclient: _renew_subs 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.743+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb884142c50 con 0x7fb88409adf0 2023-09-29T23:13:39.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7fb88c6fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7fb88c6fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7fb88c6fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8840985f0 con 0x7fb88409adf0 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 -- start start 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient: init 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient: _add_conns ranks=[1,3,7,4,2,8,5,0,6] 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-f con 0x7f2774143050 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 0x7f2774146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-g con 0x7f2774142bd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 0x7f2774146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): picked mon.noname-i con 0x7f2774143b10 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2774127a10 con 0x7f2774143050 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2774094370 con 0x7f2774142bd0 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2774095790 con 0x7f2774143b10 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.744+0000 7f2779f88700 10 monclient(hunting): authenticate will time out at 1129.104863s 2023-09-29T23:13:39.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 0x7f2774146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:60262/0 (socket says 172.21.15.12:60262) 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 0x7f2774146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:56244/0 (socket says 172.21.15.12:56244) 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 -- 172.21.15.12:0/2841222270 learned_addr learned my addr 172.21.15.12:0/2841222270 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): get_auth_request con 0x7f2774143050 auth_method 0 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): get_auth_request con 0x7f2774143b10 auth_method 0 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 0x7f2774146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): get_auth_request con 0x7f2774142bd0 auth_method 0 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2773fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 -- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 msgr2=0x7f2774146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 0x7f2774146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 -- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 msgr2=0x7f2774146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 0x7f2774146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 monclient: found mon.noname-f 2023-09-29T23:13:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 -- 172.21.15.12:0/2841222270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774143420 con 0x7f2774143050 2023-09-29T23:13:39.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f2772ffd700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 0x7f2774146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fb880011600 con 0x7fb88409adf0 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7fb87a7fc700 1 --2- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 0x7fb868038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb8800470b0 con 0x7fb88409adf0 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7fb889c97700 1 --2- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 0x7fb868038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7fb889c97700 10 monclient: get_auth_request con 0x7fb868036850 auth_method 0 2023-09-29T23:13:39.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7fb889c97700 1 --2- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 0x7fb868038d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb874001660 tx=0x7fb874007b00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.745+0000 7f27737fe700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2764001120 tx=0x7f2764015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f27640018f0 con 0x7f2774143050 2023-09-29T23:13:39.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:39.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: dump: 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.798 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764007550 con 0x7f2774143050 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 5 monclient: authenticate success, global_id 4294 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 10 monclient: _renew_subs 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277406f3c0 con 0x7f2774143050 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f276401e430 con 0x7f2774143050 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f275b7fe700 10 monclient: dump: 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.800 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 10 monclient: _renew_subs 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2774094a50 con 0x7f2774143050 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.746+0000 7f2771ffb700 4 set_mon_vals no callback set 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2764007070 con 0x7f2774143050 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f2779f88700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f2779f88700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f2779f88700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277406e960 con 0x7f2774143050 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f275b7fe700 1 --2- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 0x7f2760038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f27640077e0 con 0x7f2774143050 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.747+0000 7f2772ffd700 1 --2- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 0x7f2760038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.748+0000 7f2772ffd700 10 monclient: get_auth_request con 0x7f2760036880 auth_method 0 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.748+0000 7f2772ffd700 1 --2- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 0x7f2760038d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2774094880 tx=0x7f2768007d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.751+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8800118b0 con 0x7fb88409adf0 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.751+0000 7fb87a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.751+0000 7fb87a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.753+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27640273c0 con 0x7f2774143050 2023-09-29T23:13:39.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.753+0000 7f275b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.753+0000 7f275b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.784+0000 7fe8d1c9e700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.784+0000 7fe8d1c9e700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.784+0000 7fe8d1c9e700 10 monclient: monmap: 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.785763+0000 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.785763+0000 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.805 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.806 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.806 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.806 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.806 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.785+0000 7f8b6e5a7700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.785+0000 7f8b6e5a7700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.785+0000 7f8b6e5a7700 10 monclient: monmap: 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.786435+0000 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.786435+0000 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.786+0000 7f8b6e5a7700 1 Processor -- start 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.786+0000 7f8b6e5a7700 1 -- start start 2023-09-29T23:13:39.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.786+0000 7f8b6e5a7700 10 monclient: init 2023-09-29T23:13:39.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 Processor -- start 2023-09-29T23:13:39.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 -- start start 2023-09-29T23:13:39.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient: init 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient: _add_conns ranks=[0,2,1,3,6,5,8,9,4,7,10] 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b6806ece0 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b6806f0b0 0x7f8b6806f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-b con 0x7f8b6806f0b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8b68138610 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b68068ed0 con 0x7f8b6806ece0 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6804c960 con 0x7f8b68138610 2023-09-29T23:13:39.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6809deb0 con 0x7f8b6806f0b0 2023-09-29T23:13:39.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b6e5a7700 10 monclient(hunting): authenticate will time out at 1129.147477s 2023-09-29T23:13:39.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b67fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8b6806ece0 0x7f8b68133740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37970/0 (socket says 172.21.15.12:37970) 2023-09-29T23:13:39.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7f8b67fff700 1 -- 172.21.15.12:0/1397544384 learned_addr learned my addr 172.21.15.12:0/1397544384 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient: _add_conns ranks=[5,3,9,4,6,1,0,8,7,10,2] 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc0a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-h con 0x7fe8cc09cf30 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09a780 0x7fe8cc0a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-g con 0x7fe8cc09a780 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8cc09d300 0x7fe8cc0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc09d300 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:39.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8cc09b4e0 con 0x7fe8cc09d300 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc09ac40 con 0x7fe8cc09a780 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc09af20 con 0x7fe8cc09cf30 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.787+0000 7fe8d1c9e700 10 monclient(hunting): authenticate will time out at 1129.147806s 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8cc09d300 0x7fe8cc0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe8cc09d300 0x7fe8cc0a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52918/0 (socket says 172.21.15.12:52918) 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 -- 172.21.15.12:0/3745606845 learned_addr learned my addr 172.21.15.12:0/3745606845 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): get_auth_request con 0x7fe8cc09d300 auth_method 0 2023-09-29T23:13:39.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149796226 0 0) 0x7f8b68068ed0 con 0x7f8b6806ece0 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): my global_id is 4383 2023-09-29T23:13:39.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b50003570 con 0x7f8b6806ece0 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071662908 0 0) 0x7f8b6804c960 con 0x7f8b68138610 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): my global_id is 4303 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b68068ed0 con 0x7f8b68138610 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 -- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 msgr2=0x7fe8cc0a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc0a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 -- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09a780 msgr2=0x7fe8cc0a1600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09a780 0x7fe8cc0a1600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 -- 172.21.15.12:0/3745606845 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc068ed0 con 0x7fe8cc09d300 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365216888 0 0) 0x7f8b50003570 con 0x7f8b6806ece0 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b6806f0b0 msgr2=0x7f8b6806f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8b6806f0b0 0x7f8b6806f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 >> v1:172.21.15.153:6789/0 conn(0x7f8b68138610 legacy=0x7f8b6813a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:13:39.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6809f290 con 0x7f8b6806ece0 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f8b580018f0 con 0x7f8b6806ece0 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: dump: 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.820 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 >> v1:172.21.15.12:6789/0 conn(0x7f8b6806ece0 legacy=0x7f8b68133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient: _add_conns ranks=[3,5,7,4,2,6,1,0,8] 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b50004810 0x7f8b50004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): picked mon.g con 0x7f8b50004810 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8b50005120 0x7f8b50009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): picked mon.h con 0x7f8b50005120 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 0x7f8b5000c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): picked mon.i con 0x7f8b50009f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:39.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6809deb0 con 0x7f8b50004810 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5000cb60 con 0x7f8b50005120 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5000ccd0 con 0x7f8b50009f90 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b66ffd700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:13:39.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7fe8cbfff700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8cc09d300 0x7fe8cc0a1b40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009f20 tx=0x7fe8c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4767f0e0d5b488 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b677fe700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8b50005120 0x7f8b50009a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b677fe700 10 monclient(hunting): get_auth_request con 0x7f8b50005120 auth_method 0 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b677fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b6cb44700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 0x7f8b5000c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b6cb44700 10 monclient(hunting): get_auth_request con 0x7f8b50009f90 auth_method 0 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b6cb44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b6cb44700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.788+0000 7f8b6cb44700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b50004810 0x7f8b50004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): get_auth_request con 0x7f8b50004810 auth_method 0 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 1 -- 172.21.15.12:0/3745606845 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8c400e070 con 0x7fe8cc09d300 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: dump: 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.826 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 1 -- 172.21.15.12:0/3745606845 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c4007400 con 0x7fe8cc09d300 2023-09-29T23:13:39.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 1 -- 172.21.15.12:0/3745606845 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8c40092e0 con 0x7fe8cc09d300 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8ca7fc700 10 monclient: dump: 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.828 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.829 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 5 monclient: authenticate success, global_id 4392 2023-09-29T23:13:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 4 set_mon_vals no callback set 2023-09-29T23:13:39.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b677fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient(hunting): handle_auth_done global_id 4383 payload 293 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 -- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8b50005120 msgr2=0x7f8b50009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8b50005120 0x7f8b50009a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 -- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b50004810 msgr2=0x7f8b50004be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b50004810 0x7f8b50004be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 monclient: found mon.i 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 -- 172.21.15.12:0/1397544384 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6809f290 con 0x7f8b50009f90 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 -- 172.21.15.12:0/1397544384 reap_dead start 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6e5a7700 5 monclient: authenticate success, global_id 4383 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6e5a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6cb44700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 0x7f8b5000c370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b6809fd40 tx=0x7f8b600017a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=80f57bdf655b3b48 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8b60001ab0 con 0x7f8b50009f90 2023-09-29T23:13:39.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: dump: 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6000cd30 con 0x7f8b50009f90 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 1 -- 172.21.15.12:0/1397544384 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8b6000cea0 con 0x7f8b50009f90 2023-09-29T23:13:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b66ffd700 10 monclient: dump: 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6e5a7700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.837 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7f8b6e5a7700 4 set_mon_vals no callback set 2023-09-29T23:13:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 10 monclient: shutdown 2023-09-29T23:13:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3745606845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8cc09d300 msgr2=0x7fe8cc0a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.789+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8cc09d300 0x7fe8cc0a1b40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009f20 tx=0x7fe8c4009840 comp rx=0 tx=0).stop 2023-09-29T23:13:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 10 monclient: shutdown 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 msgr2=0x7f8b5000c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 0x7f8b5000c370 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b6809fd40 tx=0x7f8b600017a0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/1397544384 shutdown_connections 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/1397544384 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8b50009f90 0x7f8b5000c370 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/1397544384 >> 172.21.15.12:0/1397544384 conn(0x7f8b6806aab0 msgr2=0x7f8b6806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.790+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/1397544384 shutdown_connections 2023-09-29T23:13:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3745606845 shutdown_connections 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8cc09d300 0x7fe8cc0a1b40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc0a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3745606845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8cc09a780 0x7fe8cc0a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3745606845 >> 172.21.15.12:0/3745606845 conn(0x7fe8cc063d70 msgr2=0x7fe8cc09fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/1397544384 wait complete. 2023-09-29T23:13:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7f8b6e5a7700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3745606845 shutdown_connections 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7f8b6e5a7700 10 monclient: monmap: 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.791+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3745606845 wait complete. 2023-09-29T23:13:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.792+0000 7f8b6e5a7700 1 Processor -- start 2023-09-29T23:13:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.792+0000 7fe8d1c9e700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.792+0000 7fe8d1c9e700 10 monclient: monmap: 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 Processor -- start 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.792+0000 7f8b6e5a7700 1 -- start start 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.792+0000 7f8b6e5a7700 10 monclient: init 2023-09-29T23:13:39.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient: _add_conns ranks=[8,0,1,5,4,2,7,3,6] 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-e con 0x7f8b50009f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b681376f0 0x7f8b68098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-a con 0x7f8b681376f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b68098760 0x7f8b68098b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): picked mon.noname-f con 0x7f8b68098760 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680990d0 con 0x7f8b681376f0 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6809e810 con 0x7f8b68098760 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6809ebe0 con 0x7f8b50009f90 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b6e5a7700 10 monclient(hunting): authenticate will time out at 1129.153141s 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 -- start start 2023-09-29T23:13:39.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient: init 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55354/0 (socket says 172.21.15.12:55354) 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 -- 172.21.15.12:0/2143459306 learned_addr learned my addr 172.21.15.12:0/2143459306 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): get_auth_request con 0x7f8b50009f90 auth_method 0 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 -- 172.21.15.12:0/2143459306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b68098760 msgr2=0x7f8b68098b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b68098760 0x7f8b68098b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 -- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b681376f0 msgr2=0x7f8b68098220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b681376f0 0x7f8b68098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 monclient: found mon.noname-e 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 -- 172.21.15.12:0/2143459306 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68134760 con 0x7f8b50009f90 2023-09-29T23:13:39.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7f8b67fff700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b58007390 tx=0x7f8b58001690 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient: _add_conns ranks=[5,1,6,8,4,2,7,3,0] 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc11b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-h con 0x7fe8cc09cf30 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-f con 0x7fe8cc09d300 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe8cc11cfd0 0x7fe8cc119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): picked mon.noname-d con 0x7fe8cc11cfd0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc099900 con 0x7fe8cc09d300 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc11aee0 con 0x7fe8cc09cf30 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc11ba80 con 0x7fe8cc11cfd0 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.793+0000 7fe8d1c9e700 10 monclient(hunting): authenticate will time out at 1129.153904s 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:60280/0 (socket says 172.21.15.12:60280) 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 -- 172.21.15.12:0/3479972211 learned_addr learned my addr 172.21.15.12:0/3479972211 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe8cc11cfd0 0x7fe8cc119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): get_auth_request con 0x7fe8cc09d300 auth_method 0 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): get_auth_request con 0x7fe8cc11cfd0 auth_method 0 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc11b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 10 monclient(hunting): get_auth_request con 0x7fe8cc09cf30 auth_method 0 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 -- 172.21.15.12:0/3479972211 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe8cc11cfd0 msgr2=0x7fe8cc119ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe8cc11cfd0 0x7fe8cc119ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 -- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 msgr2=0x7fe8cc11b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc11b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 monclient: found mon.noname-f 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 -- 172.21.15.12:0/3479972211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c4009ae0 con 0x7fe8cc09d300 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8cb7fe700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc11b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8b58018030 con 0x7f8b50009f90 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: dump: 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 5 monclient: authenticate success, global_id 4256 2023-09-29T23:13:39.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5800fd50 con 0x7f8b50009f90 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8b5800fec0 con 0x7f8b50009f90 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b667fc700 4 set_mon_vals no callback set 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b4ffff700 10 monclient: dump: 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.859 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 10 monclient: _renew_subs 2023-09-29T23:13:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b68135650 con 0x7f8b50009f90 2023-09-29T23:13:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 10 monclient: _renew_subs 2023-09-29T23:13:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:39.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b68099ae0 con 0x7f8b50009f90 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.794+0000 7fe8caffd700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00b5e0 tx=0x7fe8bc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8bc00f990 con 0x7fe8cc09d300 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: dump: 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.863 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 5 monclient: authenticate success, global_id 4312 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8bc00faf0 con 0x7fe8cc09d300 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8bc00fc60 con 0x7fe8cc09d300 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8bb7fe700 10 monclient: dump: 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.864 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.865 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 10 monclient: _renew_subs 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8c9ffb700 4 set_mon_vals no callback set 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc04c960 con 0x7fe8cc09d300 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 10 monclient: _renew_subs 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc11c020 con 0x7fe8cc09d300 2023-09-29T23:13:39.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fa9224c3700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.795+0000 7fa9224c3700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7fa9224c3700 10 monclient: monmap: 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.796860+0000 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.796860+0000 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.867 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.868 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8b5801b070 con 0x7f8b50009f90 2023-09-29T23:13:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7f8b4ffff700 1 --2- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 0x7f8b54038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8b58019070 con 0x7f8b50009f90 2023-09-29T23:13:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7f8b677fe700 1 --2- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 0x7f8b54038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7f8b677fe700 10 monclient: get_auth_request con 0x7f8b54036880 auth_method 0 2023-09-29T23:13:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7fa9224c3700 1 Processor -- start 2023-09-29T23:13:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7fa9224c3700 1 -- start start 2023-09-29T23:13:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.796+0000 7fa9224c3700 10 monclient: init 2023-09-29T23:13:39.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7f8b677fe700 1 --2- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 0x7f8b54038d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8b680a3e20 tx=0x7f8b5c009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient: _add_conns ranks=[10,1,4,2,9,7,8,6,5,0,3] 2023-09-29T23:13:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 0x7fa91c0969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-f con 0x7fa91c06dc50 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa91c06e020 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 0x7fa91c1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-c con 0x7fa91c1365d0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c068ed0 con 0x7fa91c06e020 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c09e0a0 con 0x7fa91c06dc50 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c09eac0 con 0x7fa91c1365d0 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa9224c3700 10 monclient(hunting): authenticate will time out at 1129.157443s 2023-09-29T23:13:39.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 0x7fa91c0969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 0x7fa91c1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 0x7fa91c0969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:60286/0 (socket says 172.21.15.12:60286) 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 1 -- 172.21.15.12:0/4119549447 learned_addr learned my addr 172.21.15.12:0/4119549447 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): get_auth_request con 0x7fa91c06dc50 auth_method 0 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): get_auth_request con 0x7fa91c1365d0 auth_method 0 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 1 -- 172.21.15.12:0/4119549447 >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 msgr2=0x7fa91c0969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 1 --2- 172.21.15.12:0/4119549447 >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 0x7fa91c0969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa920a60700 1 -- 172.21.15.12:0/4119549447 >> v1:172.21.15.153:6789/0 conn(0x7fa91c06e020 legacy=0x7fa91c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 1 monclient: found mon.noname-c 2023-09-29T23:13:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.797+0000 7fa91affd700 1 -- 172.21.15.12:0/4119549447 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527455445 0 0) 0x7fa91c068ed0 con 0x7fa91c06e020 2023-09-29T23:13:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 1 -- 172.21.15.12:0/4119549447 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c04c960 con 0x7fa91c1365d0 2023-09-29T23:13:39.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa91affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8d1c9e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8d1c9e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8d1c9e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc11c420 con 0x7fe8cc09d300 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe8bc00f510 con 0x7fe8cc09d300 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8bb7fe700 1 --2- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 0x7fe8b0038c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe8bc046580 con 0x7fe8cc09d300 2023-09-29T23:13:39.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8cb7fe700 1 --2- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 0x7fe8b0038c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fe8cb7fe700 10 monclient: get_auth_request con 0x7fe8b0036750 auth_method 0 2023-09-29T23:13:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7f8b6e5a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7f8b6e5a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7f8b6e5a7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:39.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fa920a60700 1 --2- 172.21.15.12:0/4119549447 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 0x7fa91c1389b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa914009e90 tx=0x7fa914009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1fd1d3bd9201d137 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68070bc0 con 0x7f8b50009f90 2023-09-29T23:13:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 1 -- 172.21.15.12:0/4119549447 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa91400e070 con 0x7fa91c1365d0 2023-09-29T23:13:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:39.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fe8cb7fe700 1 --2- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 0x7fe8b0038c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009b10 tx=0x7fe8c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: dump: 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.880 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.881 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa9224c3700 5 monclient: authenticate success, global_id 4351 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 1 -- 172.21.15.12:0/4119549447 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91400bbf0 con 0x7fa91c1365d0 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa9224c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 1 -- 172.21.15.12:0/4119549447 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa91400bd60 con 0x7fa91c1365d0 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa9224c3700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa9224c3700 4 set_mon_vals no callback set 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fa91affd700 10 monclient: dump: 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.883 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 10 monclient: shutdown 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 -- 172.21.15.12:0/4119549447 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 msgr2=0x7fa91c1389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4119549447 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 0x7fa91c1389b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa914009e90 tx=0x7fa914009780 comp rx=0 tx=0).stop 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 -- 172.21.15.12:0/4119549447 shutdown_connections 2023-09-29T23:13:39.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4119549447 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa91c1365d0 0x7fa91c1389b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4119549447 >> v2:172.21.15.153:3300/0 conn(0x7fa91c06dc50 0x7fa91c0969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 -- 172.21.15.12:0/4119549447 >> 172.21.15.12:0/4119549447 conn(0x7fa91c06a370 msgr2=0x7fa91c0a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 -- 172.21.15.12:0/4119549447 shutdown_connections 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 1 -- 172.21.15.12:0/4119549447 wait complete. 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fa9224c3700 10 monclient: monmap: 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fd060963700 10 monclient: get_monmap_and_config 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fd060963700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fd060963700 10 monclient: monmap: 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:39.799101+0000 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:39.799101+0000 2023-09-29T23:13:39.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.888 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.798+0000 7fd060963700 1 Processor -- start 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 -- start start 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient: init 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient: _add_conns ranks=[2,10,6,9,4,3,5,1,8,7,0] 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd05c09cb70 0x7fd05c09b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): picked mon.noname-b con 0x7fd05c09cb70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 0x7fd05c09d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): picked mon.noname-f con 0x7fd05c09cf40 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): picked mon.noname-d con 0x7fd05c08d0c0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c09e560 con 0x7fd05c09cf40 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c09a620 con 0x7fd05c09cb70 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c099dd0 con 0x7fd05c08d0c0 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.799+0000 7fd060963700 10 monclient(hunting): authenticate will time out at 1129.159923s 2023-09-29T23:13:39.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05ad9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05ad9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56278/0 (socket says 172.21.15.12:56278) 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05ad9d700 1 -- 172.21.15.12:0/415929074 learned_addr learned my addr 172.21.15.12:0/415929074 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05ad9d700 10 monclient(hunting): get_auth_request con 0x7fd05c08d0c0 auth_method 0 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd05a59c700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd05c09cb70 0x7fd05c09b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.800+0000 7fd059d9b700 1 --2- 172.21.15.12:0/415929074 >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 0x7fd05c09d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): get_auth_request con 0x7fd05c09cb70 auth_method 0 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fa9224c3700 1 Processor -- start 2023-09-29T23:13:39.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 -- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd05c09cb70 msgr2=0x7fd05c09b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd05c09cb70 0x7fd05c09b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 -- 172.21.15.12:0/415929074 >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 msgr2=0x7fd05c09d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 --2- 172.21.15.12:0/415929074 >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 0x7fd05c09d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 monclient: found mon.noname-d 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 -- 172.21.15.12:0/415929074 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c068ed0 con 0x7fd05c08d0c0 2023-09-29T23:13:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd059d9b700 10 monclient: get_auth_request con 0x7fd05c09cf40 auth_method 0 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd059d9b700 1 --2- 172.21.15.12:0/415929074 >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 0x7fd05c09d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05ad9d700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd05400bfc0 tx=0x7fd05400d040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5a254acca40fe953 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 1 -- 172.21.15.12:0/415929074 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd05400b2e0 con 0x7fd05c08d0c0 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: dump: 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd060963700 5 monclient: authenticate success, global_id 4326 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 1 -- 172.21.15.12:0/415929074 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05400b480 con 0x7fd05c08d0c0 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd060963700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd060963700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd060963700 4 set_mon_vals no callback set 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 1 -- 172.21.15.12:0/415929074 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd054009a90 con 0x7fd05c08d0c0 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fd05959a700 10 monclient: dump: 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.898 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fa9224c3700 1 -- start start 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.801+0000 7fa9224c3700 10 monclient: init 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient: _add_conns ranks=[5,0,1,6,2,3,4,8,7] 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-h con 0x7fa91c06dc50 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa91c06e020 0x7fa91c1359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-a con 0x7fa91c06e020 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 0x7fa91c121a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): picked mon.noname-f con 0x7fa91c1365d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c09e280 con 0x7fa91c06e020 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c123cb0 con 0x7fa91c1365d0 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c124b30 con 0x7fa91c06dc50 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa9224c3700 10 monclient(hunting): authenticate will time out at 1129.162207s 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 0x7fa91c121a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:39910/0 (socket says 172.21.15.12:39910) 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 1 -- 172.21.15.12:0/4148916922 learned_addr learned my addr 172.21.15.12:0/4148916922 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): get_auth_request con 0x7fa91c06dc50 auth_method 0 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): get_auth_request con 0x7fa91c1365d0 auth_method 0 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 -- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 msgr2=0x7fa91c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 -- 172.21.15.12:0/4148916922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa91c06e020 msgr2=0x7fa91c1359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa91c06e020 0x7fa91c1359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 monclient: found mon.noname-f 2023-09-29T23:13:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa920a60700 1 -- 172.21.15.12:0/4148916922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009a20 con 0x7fa91c1365d0 2023-09-29T23:13:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fa91bfff700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:39.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fd060963700 10 monclient: shutdown 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fd060963700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fd060963700 1 -- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 msgr2=0x7fd05c08d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.802+0000 7fd060963700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd05400bfc0 tx=0x7fd05400d040 comp rx=0 tx=0).stop 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 -- 172.21.15.12:0/415929074 shutdown_connections 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd05c08d0c0 0x7fd05c08d510 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 --2- 172.21.15.12:0/415929074 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd05c09cb70 0x7fd05c09b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa920a60700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 0x7fa91c121a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa914016000 tx=0x7fa914019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 --2- 172.21.15.12:0/415929074 >> v2:172.21.15.153:3300/0 conn(0x7fd05c09cf40 0x7fd05c09d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 -- 172.21.15.12:0/415929074 >> 172.21.15.12:0/415929074 conn(0x7fd05c0699a0 msgr2=0x7fd05c09ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:39.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 -- 172.21.15.12:0/415929074 shutdown_connections 2023-09-29T23:13:39.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 1 -- 172.21.15.12:0/415929074 wait complete. 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa914012be0 con 0x7fa91c1365d0 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: dump: 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.909 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914012d80 con 0x7fa91c1365d0 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa91401bcf0 con 0x7fa91c1365d0 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:39.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa91a7fc700 4 set_mon_vals no callback set 2023-09-29T23:13:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 10 monclient: build_initial_monmap 2023-09-29T23:13:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8bc00f7c0 con 0x7fe8cc09d300 2023-09-29T23:13:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fe8bb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fe8bb7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fd060963700 10 monclient: monmap: 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:39.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fd060963700 1 Processor -- start 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fd060963700 1 -- start start 2023-09-29T23:13:39.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fd060963700 10 monclient: init 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.803+0000 7fa903fff700 10 monclient: dump: 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 5 monclient: authenticate success, global_id 4339 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 10 monclient: _renew_subs 2023-09-29T23:13:39.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c124e10 con 0x7fa91c1365d0 2023-09-29T23:13:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 10 monclient: _renew_subs 2023-09-29T23:13:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.804+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c146780 con 0x7fa91c1365d0 2023-09-29T23:13:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient: _add_conns ranks=[3,0,8,7,6,4,5,2,1] 2023-09-29T23:13:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd05c08d0c0 0x7fd05c091ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): picked mon.noname-g con 0x7fd05c08d0c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): picked mon.noname-a con 0x7fd05c09cb70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd05c09cf40 0x7fd05c092960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): picked mon.noname-e con 0x7fd05c09cf40 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c09b9e0 con 0x7fd05c09cb70 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c099f70 con 0x7fd05c08d0c0 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0956a0 con 0x7fd05c09cf40 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd060963700 10 monclient(hunting): authenticate will time out at 1129.165185s 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52952/0 (socket says 172.21.15.12:52952) 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 -- 172.21.15.12:0/2988389521 learned_addr learned my addr 172.21.15.12:0/2988389521 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): get_auth_request con 0x7fd05c09cb70 auth_method 0 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd05c09cf40 0x7fd05c092960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd05c08d0c0 0x7fd05c091ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): get_auth_request con 0x7fd05c09cf40 auth_method 0 2023-09-29T23:13:39.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): get_auth_request con 0x7fd05c08d0c0 auth_method 0 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd05ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 -- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd05c09cf40 msgr2=0x7fd05c092960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd05c09cf40 0x7fd05c092960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 -- 172.21.15.12:0/2988389521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd05c08d0c0 msgr2=0x7fd05c091ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd05c08d0c0 0x7fd05c091ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 monclient: found mon.noname-a 2023-09-29T23:13:39.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.805+0000 7fd059d9b700 1 -- 172.21.15.12:0/2988389521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05400bb00 con 0x7fd05c09cb70 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd059d9b700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd04400c810 tx=0x7fd044006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd044006e00 con 0x7fd05c09cb70 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: dump: 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:39.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd060963700 5 monclient: authenticate success, global_id 4401 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044011a10 con 0x7fd05c09cb70 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd044012420 con 0x7fd05c09cb70 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd058d99700 4 set_mon_vals no callback set 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7fd0527fc700 10 monclient: dump: 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:39.926 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:39.927 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:39.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b5800f8d0 con 0x7f8b50009f90 2023-09-29T23:13:39.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7f8b4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.806+0000 7f8b4ffff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 10 monclient: _renew_subs 2023-09-29T23:13:39.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c04c960 con 0x7fd05c09cb70 2023-09-29T23:13:39.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 10 monclient: _renew_subs 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c095330 con 0x7fd05c09cb70 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.807+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd044011b80 con 0x7fd05c09cb70 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd0527fc700 1 --2- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 0x7fd048039240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd060963700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd044047040 con 0x7fd05c09cb70 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd060963700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd060963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c066f20 con 0x7fd05c09cb70 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd05a59c700 1 --2- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 0x7fd048039240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd05a59c700 10 monclient: get_auth_request con 0x7fd048036d90 auth_method 0 2023-09-29T23:13:39.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.808+0000 7fd05a59c700 1 --2- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 0x7fd048039240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00efd0 tx=0x7fd04c010040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd044012580 con 0x7fd05c09cb70 2023-09-29T23:13:39.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fd0527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fd0527fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa91401b3f0 con 0x7fa91c1365d0 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fa903fff700 1 --2- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 0x7fa908038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa914048c80 con 0x7fa91c1365d0 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fa91bfff700 1 --2- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 0x7fa908038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.813+0000 7fa91bfff700 10 monclient: get_auth_request con 0x7fa908036850 auth_method 0 2023-09-29T23:13:39.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.814+0000 7fa91bfff700 1 --2- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 0x7fa908038d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa90c011540 tx=0x7fa90c009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.818+0000 7fa9224c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.818+0000 7fa9224c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.818+0000 7fa9224c3700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.818+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c06ebf0 con 0x7fa91c1365d0 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.823+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa91402e9a0 con 0x7fa91c1365d0 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.823+0000 7fa903fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:39.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:39.823+0000 7fa903fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.018+0000 7f8b6e5a7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.018+0000 7f8b6e5a7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.018+0000 7f8b6e5a7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.018+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8b6809f570 con 0x7f8b50009f90 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.019+0000 7f8b4ffff700 1 -- 172.21.15.12:0/2143459306 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8b5802d090 con 0x7f8b50009f90 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.019+0000 7f8b4ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.019+0000 7f8b4ffff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.020 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 msgr2=0x7f8b54038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 0x7f8b54038d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8b680a3e20 tx=0x7f8b5c009060 comp rx=0 tx=0).stop 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 10 monclient: shutdown 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 msgr2=0x7f8b68134ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b58007390 tx=0x7f8b58001690 comp rx=0 tx=0).stop 2023-09-29T23:13:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 shutdown_connections 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> v2:172.21.15.153:6812/83257 conn(0x7f8b54036880 0x7f8b54038d30 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8b680a3e20 tx=0x7f8b5c009060 comp rx=0 tx=0).stop 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b68098760 0x7f8b68098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8b50009f90 0x7f8b68134ee0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 --2- 172.21.15.12:0/2143459306 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8b681376f0 0x7f8b68098220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.022+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 >> 172.21.15.12:0/2143459306 conn(0x7f8b6806aab0 msgr2=0x7f8b6806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.023+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 shutdown_connections 2023-09-29T23:13:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7f8b6e5a7700 1 -- 172.21.15.12:0/2143459306 wait complete. 2023-09-29T23:13:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd060963700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd060963700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd060963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd0400050d0 con 0x7fd05c09cb70 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd0527fc700 1 -- 172.21.15.12:0/2988389521 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd044019ac0 con 0x7fd05c09cb70 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd0527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.025+0000 7fd0527fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.026 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 msgr2=0x7fd048039240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 0x7fd048039240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00efd0 tx=0x7fd04c010040 comp rx=0 tx=0).stop 2023-09-29T23:13:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 10 monclient: shutdown 2023-09-29T23:13:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 msgr2=0x7fd05c092420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd04400c810 tx=0x7fd044006d70 comp rx=0 tx=0).stop 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 shutdown_connections 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd05c09cf40 0x7fd05c092960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> v2:172.21.15.153:6812/83257 conn(0x7fd048036d90 0x7fd048039240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd05c08d0c0 0x7fd05c091ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 --2- 172.21.15.12:0/2988389521 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd05c09cb70 0x7fd05c092420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 >> 172.21.15.12:0/2988389521 conn(0x7fd05c0699a0 msgr2=0x7fd05c06e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 shutdown_connections 2023-09-29T23:13:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.028+0000 7fd060963700 1 -- 172.21.15.12:0/2988389521 wait complete. 2023-09-29T23:13:40.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.4 2023-09-29T23:13:40.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.1 2023-09-29T23:13:40.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.127+0000 7fe8d1c9e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:40.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.127+0000 7fe8d1c9e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:40.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.127+0000 7fe8d1c9e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:40.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.127+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe8cc070bc0 con 0x7fe8cc09d300 2023-09-29T23:13:40.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.131+0000 7fe8bb7fe700 1 -- 172.21.15.12:0/3479972211 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe8bc016090 con 0x7fe8cc09d300 2023-09-29T23:13:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.131+0000 7fe8bb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:40.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.131+0000 7fe8bb7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.131 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 msgr2=0x7fe8b0038c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 0x7fe8b0038c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009b10 tx=0x7fe8c0008040 comp rx=0 tx=0).stop 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 10 monclient: shutdown 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 msgr2=0x7fe8cc119980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.133+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00b5e0 tx=0x7fe8bc0090f0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 shutdown_connections 2023-09-29T23:13:40.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> v2:172.21.15.153:6812/83257 conn(0x7fe8b0036750 0x7fe8b0038c00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb88c6fc700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb88c6fc700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe8cc11cfd0 0x7fe8cc119ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe8cc09cf30 0x7fe8cc11b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 --2- 172.21.15.12:0/3479972211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc09d300 0x7fe8cc119980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 >> 172.21.15.12:0/3479972211 conn(0x7fe8cc063d70 msgr2=0x7fe8cc09db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb88c6fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:40.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb884070bc0 con 0x7fb88409adf0 2023-09-29T23:13:40.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 shutdown_connections 2023-09-29T23:13:40.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fe8d1c9e700 1 -- 172.21.15.12:0/3479972211 wait complete. 2023-09-29T23:13:40.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb87a7fc700 1 -- 172.21.15.12:0/365538434 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb88000fac0 con 0x7fb88409adf0 2023-09-29T23:13:40.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb87a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:40.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.136+0000 7fb87a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.139 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.139+0000 7f2779f88700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:40.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.139+0000 7f2779f88700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:40.140 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.139+0000 7f2779f88700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:40.140 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.139+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2774066f20 con 0x7f2774143050 2023-09-29T23:13:40.140 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 msgr2=0x7fb868038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.140 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 0x7fb868038d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb874001660 tx=0x7fb874007b00 comp rx=0 tx=0).stop 2023-09-29T23:13:40.140 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 10 monclient: shutdown 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 msgr2=0x7fb884143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb88000c800 tx=0x7fb880008040 comp rx=0 tx=0).stop 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7f275b7fe700 1 -- 172.21.15.12:0/2841222270 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2764017030 con 0x7f2774143050 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7f275b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:40.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7f275b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.142 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 shutdown_connections 2023-09-29T23:13:40.142 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb884143a70 0x7fb8841465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.142 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.142 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb88409adf0 0x7fb884143060 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> v2:172.21.15.153:6812/83257 conn(0x7fb868036850 0x7fb868038d00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 --2- 172.21.15.12:0/365538434 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb88409d5a0 0x7fb884142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.140+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 >> 172.21.15.12:0/365538434 conn(0x7fb88406a370 msgr2=0x7fb88406d360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.141+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 shutdown_connections 2023-09-29T23:13:40.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.141+0000 7fb88c6fc700 1 -- 172.21.15.12:0/365538434 wait complete. 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 msgr2=0x7f2760038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 0x7f2760038d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2774094880 tx=0x7f2768007d10 comp rx=0 tx=0).stop 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 10 monclient: shutdown 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 msgr2=0x7f2774142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2764001120 tx=0x7f2764015000 comp rx=0 tx=0).stop 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 shutdown_connections 2023-09-29T23:13:40.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2774143b10 0x7f2774146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2774143050 0x7f2774142690 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> v2:172.21.15.153:6812/83257 conn(0x7f2760036880 0x7f2760038d30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 --2- 172.21.15.12:0/2841222270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2774142bd0 0x7f2774146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 >> 172.21.15.12:0/2841222270 conn(0x7f277409ee30 msgr2=0x7f2774097ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 shutdown_connections 2023-09-29T23:13:40.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.142+0000 7f2779f88700 1 -- 172.21.15.12:0/2841222270 wait complete. 2023-09-29T23:13:40.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa9224c3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:40.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa9224c3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:40.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa9224c3700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:40.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa91c070bc0 con 0x7fa91c1365d0 2023-09-29T23:13:40.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa903fff700 1 -- 172.21.15.12:0/4148916922 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa9140253a0 con 0x7fa91c1365d0 2023-09-29T23:13:40.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa903fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:40.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.150+0000 7fa903fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:40.151 INFO:teuthology.orchestra.run.smithi012.stdout:47244640257 2023-09-29T23:13:40.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.2 2023-09-29T23:13:40.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 msgr2=0x7fa908038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 0x7fa908038d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa90c011540 tx=0x7fa90c009250 comp rx=0 tx=0).stop 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 10 monclient: shutdown 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 msgr2=0x7fa91c121a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 0x7fa91c121a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa914016000 tx=0x7fa914019000 comp rx=0 tx=0).stop 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 shutdown_connections 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> v2:172.21.15.153:6812/83257 conn(0x7fa908036850 0x7fa908038d00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa91c1365d0 0x7fa91c121a20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa91c06dc50 0x7fa91c09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 --2- 172.21.15.12:0/4148916922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa91c06e020 0x7fa91c1359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:40.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 >> 172.21.15.12:0/4148916922 conn(0x7fa91c06a370 msgr2=0x7fa91c138120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:40.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 shutdown_connections 2023-09-29T23:13:40.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:40.153+0000 7fa9224c3700 1 -- 172.21.15.12:0/4148916922 wait complete. 2023-09-29T23:13:40.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.3 2023-09-29T23:13:40.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2023-09-29T23:13:40.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.5 2023-09-29T23:13:41.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-29T23:13:41.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:13:41.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:13:41.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-29T23:13:41.161 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:13:41.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-29T23:13:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.174+0000 7fb1eafb3700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.174+0000 7fb1eafb3700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.174+0000 7fb1eafb3700 10 monclient: monmap: 2023-09-29T23:13:41.175 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.175336+0000 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.175336+0000 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.176 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 Processor -- start 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 -- start start 2023-09-29T23:13:41.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient: init 2023-09-29T23:13:41.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient: _add_conns ranks=[0,3,1,7,4,10,9,2,8,5,6] 2023-09-29T23:13:41.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1e406eff0 addr v1:172.21.15.12:6789/0 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 0x7fb1e406f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-g con 0x7fb1e406f3c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb1e4136370 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e4068ed0 con 0x7fb1e406eff0 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e404c960 con 0x7fb1e4136370 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40a0be0 con 0x7fb1e406f3c0 2023-09-29T23:13:41.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1eafb3700 10 monclient(hunting): authenticate will time out at 1130.535915s 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.175+0000 7fb1e8d4f700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fb1e406eff0 0x7fb1e4131860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37972/0 (socket says 172.21.15.12:37972) 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e8d4f700 1 -- 172.21.15.12:0/2704063621 learned_addr learned my addr 172.21.15.12:0/2704063621 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 --2- 172.21.15.12:0/2704063621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 0x7fb1e406f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): get_auth_request con 0x7fb1e406f3c0 auth_method 0 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074801111 0 0) 0x7fb1e4068ed0 con 0x7fb1e406eff0 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): my global_id is 4410 2023-09-29T23:13:41.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d4003570 con 0x7fb1e406eff0 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378075819 0 0) 0x7fb1e404c960 con 0x7fb1e4136370 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 10 monclient(hunting): my global_id is 4348 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4068ed0 con 0x7fb1e4136370 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 -- 172.21.15.12:0/2704063621 >> v1:172.21.15.153:6789/0 conn(0x7fb1e4136370 legacy=0x7fb1e4138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 -- 172.21.15.12:0/2704063621 >> v1:172.21.15.12:6789/0 conn(0x7fb1e406eff0 legacy=0x7fb1e4131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 monclient: found mon.noname-g 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.153:3301/0 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 -- 172.21.15.12:0/2704063621 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e406de00 con 0x7fb1e406f3c0 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.176+0000 7fb1e3fff700 1 --2- 172.21.15.12:0/2704063621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 0x7fb1e406f790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009910 tx=0x7fb1d00091a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=b4b4f878d7788487 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 <== mon.3 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1d0011ab0 con 0x7fb1e406f3c0 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: dump: 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1eafb3700 5 monclient: authenticate success, global_id 4269 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 <== mon.3 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d0011c30 con 0x7fb1e406f3c0 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1eafb3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1eafb3700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 1 -- 172.21.15.12:0/2704063621 <== mon.3 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1d0011da0 con 0x7fb1e406f3c0 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1eafb3700 4 set_mon_vals no callback set 2023-09-29T23:13:41.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.177+0000 7fb1e37fe700 10 monclient: dump: 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 10 monclient: shutdown 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2704063621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 msgr2=0x7fb1e406f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2704063621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 0x7fb1e406f790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009910 tx=0x7fb1d00091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2704063621 shutdown_connections 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2704063621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1e406f3c0 0x7fb1e406f790 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2704063621 >> 172.21.15.12:0/2704063621 conn(0x7fb1e40a0ea0 msgr2=0x7fb1e40a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2704063621 shutdown_connections 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.178+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2704063621 wait complete. 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.179+0000 7fb1eafb3700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.179+0000 7fb1eafb3700 10 monclient: monmap: 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.194 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.179+0000 7fb1eafb3700 1 Processor -- start 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 -- start start 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient: init 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient: _add_conns ranks=[5,8,7,3,2,1,0,6,4] 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 0x7fb1e409dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-h con 0x7fb1e406eff0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 0x7fb1e4136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-e con 0x7fb1e406f3c0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb1e4136370 0x7fb1e4136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): picked mon.noname-i con 0x7fb1e4136370 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e409e4e0 con 0x7fb1e406eff0 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4137b20 con 0x7fb1e4136370 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40a04b0 con 0x7fb1e406f3c0 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.180+0000 7fb1eafb3700 10 monclient(hunting): authenticate will time out at 1130.540925s 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 0x7fb1e4136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 0x7fb1e4136740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55366/0 (socket says 172.21.15.12:55366) 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 1 -- 172.21.15.12:0/2154138077 learned_addr learned my addr 172.21.15.12:0/2154138077 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): get_auth_request con 0x7fb1e406f3c0 auth_method 0 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e9550700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb1e4136370 0x7fb1e4136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e9550700 10 monclient(hunting): get_auth_request con 0x7fb1e4136370 auth_method 0 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e9550700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e9550700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e9550700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 0x7fb1e409dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): get_auth_request con 0x7fb1e406eff0 auth_method 0 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 -- 172.21.15.12:0/2154138077 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 msgr2=0x7fb1e4136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 0x7fb1e4136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 -- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb1e4136370 msgr2=0x7fb1e4136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb1e4136370 0x7fb1e4136c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 monclient: found mon.noname-h 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 -- 172.21.15.12:0/2154138077 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0009420 con 0x7fb1e406eff0 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.181+0000 7fb1e8d4f700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 0x7fb1e409dfa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c560 tx=0x7fb1d8007d10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1d80070d0 con 0x7fb1e406eff0 2023-09-29T23:13:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: dump: 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8007250 con 0x7fb1e406eff0 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb1d8011360 con 0x7fb1e406eff0 2023-09-29T23:13:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e0ff9700 10 monclient: dump: 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 5 monclient: authenticate success, global_id 4253 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 10 monclient: _renew_subs 2023-09-29T23:13:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4139090 con 0x7fb1e406eff0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 10 monclient: _renew_subs 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e4138db0 con 0x7fb1e406eff0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.182+0000 7fb1e2ffd700 4 set_mon_vals no callback set 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.183+0000 7fb1eafb3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.183+0000 7fb1eafb3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.183+0000 7fb1eafb3700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.183+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c40050a0 con 0x7fb1e406eff0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.184+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fb1d8005b30 con 0x7fb1e406eff0 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.184+0000 7fb1e0ff9700 1 --2- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 0x7fb1c8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.184+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb1d8046230 con 0x7fb1e406eff0 2023-09-29T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.184+0000 7fb1e3fff700 1 --2- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 0x7fb1c8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.184+0000 7fb1e3fff700 10 monclient: get_auth_request con 0x7fb1c8036850 auth_method 0 2023-09-29T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.185+0000 7fb1e3fff700 1 --2- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 0x7fb1c8038d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00097b0 tx=0x7fb1d0017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.189+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1d800f670 con 0x7fb1e406eff0 2023-09-29T23:13:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.189+0000 7fb1e0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.189+0000 7fb1e0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.231+0000 7f99905f1700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.231+0000 7f99905f1700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.231+0000 7f99905f1700 10 monclient: monmap: 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.232451+0000 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.232451+0000 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.233 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.232+0000 7f99905f1700 1 Processor -- start 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.232+0000 7f99905f1700 1 -- start start 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.232+0000 7f99905f1700 10 monclient: init 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient: _add_conns ranks=[9,7,5,3,2,1,8,10,4,6,0] 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 0x7f998812fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-a con 0x7f998812f660 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-i con 0x7f998806eb60 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f998809d5a0 0x7f998809da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-h con 0x7f998809d5a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998806bcf0 con 0x7f998812f660 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f998806d090 con 0x7f998809d5a0 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f998812fa30 con 0x7f998806eb60 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f99905f1700 10 monclient(hunting): authenticate will time out at 1130.593611s 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998e38d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 0x7f998812fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998db8c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998db8c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:56274/0 (socket says 172.21.15.12:56274) 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998e38d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 0x7f998812fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52958/0 (socket says 172.21.15.12:52958) 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998db8c700 1 -- 172.21.15.12:0/2479349777 learned_addr learned my addr 172.21.15.12:0/2479349777 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998db8c700 10 monclient(hunting): get_auth_request con 0x7f998806eb60 auth_method 0 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998db8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.233+0000 7f998eb8e700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f998809d5a0 0x7f998809da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): get_auth_request con 0x7f998812f660 auth_method 0 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): get_auth_request con 0x7f998809d5a0 auth_method 0 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998e38d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 -- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f998809d5a0 msgr2=0x7f998809da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f998809d5a0 0x7f998809da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 -- 172.21.15.12:0/2479349777 >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 msgr2=0x7f998812fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 --2- 172.21.15.12:0/2479349777 >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 0x7f998812fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 monclient: found mon.noname-i 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 -- 172.21.15.12:0/2479349777 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988068ed0 con 0x7f998806eb60 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998db8c700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f997800ba00 tx=0x7f997800b300 comp rx=0 tx=0).ready entity=mon.7 client_cookie=b157b0c922d2b833 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998d38b700 1 -- 172.21.15.12:0/2479349777 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f997800ad10 con 0x7f998806eb60 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998d38b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998d38b700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:13:41.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.234+0000 7f998d38b700 10 monclient: dump: 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.242 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 1 -- 172.21.15.12:0/2479349777 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997800ae90 con 0x7f998806eb60 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 1 -- 172.21.15.12:0/2479349777 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f997800b000 con 0x7f998806eb60 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.235+0000 7f998d38b700 10 monclient: dump: 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 5 monclient: authenticate success, global_id 4300 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 4 set_mon_vals no callback set 2023-09-29T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 10 monclient: shutdown 2023-09-29T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 1 -- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 msgr2=0x7f998806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.236+0000 7f99905f1700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f997800ba00 tx=0x7f997800b300 comp rx=0 tx=0).stop 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 -- 172.21.15.12:0/2479349777 shutdown_connections 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 --2- 172.21.15.12:0/2479349777 >> v2:172.21.15.12:3300/0 conn(0x7f998812f660 0x7f998812fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f998809d5a0 0x7f998809da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 --2- 172.21.15.12:0/2479349777 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998806ef70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 -- 172.21.15.12:0/2479349777 >> 172.21.15.12:0/2479349777 conn(0x7f998806a370 msgr2=0x7f99880a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 -- 172.21.15.12:0/2479349777 shutdown_connections 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.237+0000 7f99905f1700 1 -- 172.21.15.12:0/2479349777 wait complete. 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.238+0000 7f99905f1700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.238+0000 7f99905f1700 10 monclient: monmap: 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.251 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.238+0000 7f99905f1700 1 Processor -- start 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.238+0000 7f99905f1700 1 -- start start 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.238+0000 7f99905f1700 10 monclient: init 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient: _add_conns ranks=[7,8,0,4,2,5,6,3,1] 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998812bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-i con 0x7f998806eb60 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f998809d5a0 0x7f998809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-e con 0x7f998809d5a0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:41.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): picked mon.noname-a con 0x7f998812f660 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9988119b20 con 0x7f998812f660 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f998811b1a0 con 0x7f998806eb60 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f998806c5b0 con 0x7f998809d5a0 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f99905f1700 10 monclient(hunting): authenticate will time out at 1130.599542s 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52964/0 (socket says 172.21.15.12:52964) 2023-09-29T23:13:41.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 1 -- 172.21.15.12:0/3854800857 learned_addr learned my addr 172.21.15.12:0/3854800857 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): get_auth_request con 0x7f998812f660 auth_method 0 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998e38d700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998812bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998eb8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998e38d700 10 monclient(hunting): get_auth_request con 0x7f998806eb60 auth_method 0 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998e38d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998e38d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.239+0000 7f998e38d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f998809d5a0 0x7f998809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): get_auth_request con 0x7f998809d5a0 auth_method 0 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998db8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 -- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f998809d5a0 msgr2=0x7f998809acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f998809d5a0 0x7f998809acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 -- 172.21.15.12:0/3854800857 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 msgr2=0x7f998812bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998812bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 monclient: found mon.noname-a 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 -- 172.21.15.12:0/3854800857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997800b540 con 0x7f998812f660 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998e38d700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998812bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f998eb8e700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9980002d30 tx=0x7f9980016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.240+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9980019040 con 0x7f998812f660 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: dump: 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99800147d0 con 0x7f998812f660 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9980017dc0 con 0x7f998812f660 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f997e7fc700 10 monclient: dump: 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 5 monclient: authenticate success, global_id 4428 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 10 monclient: _renew_subs 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f998cb8a700 4 set_mon_vals no callback set 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998804c960 con 0x7f998812f660 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 10 monclient: _renew_subs 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.241+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9988119870 con 0x7f998812f660 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f99905f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f99905f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f99905f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9988070bc0 con 0x7f998812f660 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9980003040 con 0x7f998812f660 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f997e7fc700 1 --2- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 0x7f996c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9980050010 con 0x7f998812f660 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f998e38d700 1 --2- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 0x7f996c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.243+0000 7f998e38d700 10 monclient: get_auth_request con 0x7f996c036880 auth_method 0 2023-09-29T23:13:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.244+0000 7f998e38d700 1 --2- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 0x7f996c038d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d80 tx=0x7f9984008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.252+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9980017bd0 con 0x7f998812f660 2023-09-29T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.252+0000 7f997e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.252+0000 7f997e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.312+0000 7f12cb946700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.312+0000 7f12cb946700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.312+0000 7f12cb946700 10 monclient: monmap: 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.313463+0000 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.313463+0000 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.314 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 1 Processor -- start 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 1 -- start start 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 10 monclient: init 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 10 monclient: _add_conns ranks=[4,6,3,10,5,7,0,8,9,2,1] 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f12c412f660 0x7f12c412fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-c con 0x7f12c412f660 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.313+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-d con 0x7f12c406eb60 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:41.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c40631f0 0x7f12c409d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-g con 0x7f12c40631f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c406bcf0 con 0x7f12c40631f0 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f12c406d090 con 0x7f12c412f660 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f12c412fa30 con 0x7f12c406eb60 2023-09-29T23:13:41.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12cb946700 10 monclient(hunting): authenticate will time out at 1130.674071s 2023-09-29T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56292/0 (socket says 172.21.15.12:56292) 2023-09-29T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 -- 172.21.15.12:0/1970065451 learned_addr learned my addr 172.21.15.12:0/1970065451 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): get_auth_request con 0x7f12c406eb60 auth_method 0 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2023-09-29T23:13:41.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 -- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f12c412f660 msgr2=0x7f12c412fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f12c412f660 0x7f12c412fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 -- 172.21.15.12:0/1970065451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c40631f0 msgr2=0x7f12c409d5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c40631f0 0x7f12c409d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 monclient: found mon.noname-d 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 -- 172.21.15.12:0/1970065451 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c4068ed0 con 0x7f12c406eb60 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12c8ee1700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009a10 tx=0x7f12b4009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=2526fe25b15415c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12bbfff700 1 -- 172.21.15.12:0/1970065451 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12b400bca0 con 0x7f12c406eb60 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.321 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12bbfff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.314+0000 7f12bbfff700 10 monclient: dump: 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.322 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 1 -- 172.21.15.12:0/1970065451 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b400be20 con 0x7f12c406eb60 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 1 -- 172.21.15.12:0/1970065451 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12b40118a0 con 0x7f12c406eb60 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:41.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12bbfff700 10 monclient: dump: 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.324 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 5 monclient: authenticate success, global_id 4335 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 4 set_mon_vals no callback set 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 10 monclient: shutdown 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 1 -- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 msgr2=0x7f12c406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f12cb946700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009a10 tx=0x7f12b4009250 comp rx=0 tx=0).stop 2023-09-29T23:13:41.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f120c59a700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.315+0000 7f120c59a700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.316+0000 7f120c59a700 10 monclient: monmap: 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.316865+0000 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.316865+0000 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.328 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.329 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.316+0000 7f120c59a700 1 Processor -- start 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.316+0000 7f120c59a700 1 -- start start 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.316+0000 7f120c59a700 10 monclient: init 2023-09-29T23:13:41.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient: _add_conns ranks=[9,5,10,6,8,4,3,1,7,0,2] 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 0x7f120409e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-a con 0x7f12040698a0 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f120406f050 0x7f120406f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-h con 0x7f120406f050 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f120411d9b0 0x7f12041332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-f con 0x7f120411d9b0 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1204066a20 con 0x7f12040698a0 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120406f9c0 con 0x7f120411d9b0 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1204069c70 con 0x7f120406f050 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120c59a700 10 monclient(hunting): authenticate will time out at 1130.677551s 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 0x7f120409e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 0x7f120409e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52976/0 (socket says 172.21.15.12:52976) 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 1 -- 172.21.15.12:0/1127731116 learned_addr learned my addr 172.21.15.12:0/1127731116 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): get_auth_request con 0x7f12040698a0 auth_method 0 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 1 --2- 172.21.15.12:0/1127731116 >> v2:172.21.15.153:3300/0 conn(0x7f120411d9b0 0x7f12041332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): get_auth_request con 0x7f120411d9b0 auth_method 0 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120a336700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.317+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 -- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f120406f050 msgr2=0x7f120406f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 --2- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f120406f050 0x7f120406f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 -- 172.21.15.12:0/1127731116 >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 msgr2=0x7f120409e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 --2- 172.21.15.12:0/1127731116 >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 0x7f120409e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 monclient: found mon.noname-f 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 -- 172.21.15.12:0/1127731116 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204068ed0 con 0x7f120411d9b0 2023-09-29T23:13:41.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f12cb946700 1 -- 172.21.15.12:0/1970065451 shutdown_connections 2023-09-29T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f12cb946700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c40631f0 0x7f12c409d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f12cb946700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f12c412f660 0x7f12c412fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f12cb946700 1 --2- 172.21.15.12:0/1970065451 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f12c406eb60 0x7f12c406ef70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f12cb946700 1 -- 172.21.15.12:0/1970065451 >> 172.21.15.12:0/1970065451 conn(0x7f12c406a370 msgr2=0x7f12c40a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120ab37700 1 --2- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f120411d9b0 0x7f12041332e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009e60 tx=0x7f11fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11504108927a795e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 1 -- 172.21.15.12:0/1127731116 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f11fc00e070 con 0x7f120411d9b0 2023-09-29T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 10 monclient: dump: 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.339 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.340 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120c59a700 5 monclient: authenticate success, global_id 4357 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 1 -- 172.21.15.12:0/1127731116 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc012430 con 0x7f120411d9b0 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120c59a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f1209334700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120c59a700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.318+0000 7f120c59a700 4 set_mon_vals no callback set 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.319+0000 7f120c59a700 10 monclient: shutdown 2023-09-29T23:13:41.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.319+0000 7f120c59a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.319+0000 7f120c59a700 1 -- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f120411d9b0 msgr2=0x7f12041332e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.319+0000 7f120c59a700 1 --2- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f120411d9b0 0x7f12041332e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009e60 tx=0x7f11fc009780 comp rx=0 tx=0).stop 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f120c59a700 1 -- 172.21.15.12:0/1127731116 shutdown_connections 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f120c59a700 1 --2- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f120411d9b0 0x7f12041332e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f120c59a700 1 --2- 172.21.15.12:0/1127731116 >> v2:172.21.15.12:3300/0 conn(0x7f12040698a0 0x7f120409e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f120c59a700 1 --2- 172.21.15.12:0/1127731116 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f120406f050 0x7f120406f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f120c59a700 1 -- 172.21.15.12:0/1127731116 >> 172.21.15.12:0/1127731116 conn(0x7f12040a3330 msgr2=0x7f12040a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f12cb946700 1 -- 172.21.15.12:0/1970065451 shutdown_connections 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f12cb946700 1 -- 172.21.15.12:0/1970065451 wait complete. 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f12cb946700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.321+0000 7f12cb946700 10 monclient: monmap: 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.347 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.322+0000 7f12cb946700 1 Processor -- start 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.322+0000 7f12cb946700 1 -- start start 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.322+0000 7f12cb946700 10 monclient: init 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient: _add_conns ranks=[1,0,3,8,2,4,6,7,5] 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-f con 0x7f12c40631f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12c406eb60 0x7f12c412cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-a con 0x7f12c406eb60 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 0x7f12c4099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): picked mon.noname-g con 0x7f12c412f660 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c409b650 con 0x7f12c406eb60 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c409c170 con 0x7f12c40631f0 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4127010 con 0x7f12c412f660 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12cb946700 10 monclient(hunting): authenticate will time out at 1130.683310s 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:60330/0 (socket says 172.21.15.12:60330) 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 0x7f12c4099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 -- 172.21.15.12:0/723589893 learned_addr learned my addr 172.21.15.12:0/723589893 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): get_auth_request con 0x7f12c40631f0 auth_method 0 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): get_auth_request con 0x7f12c412f660 auth_method 0 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 -- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 msgr2=0x7f12c4099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 0x7f12c4099890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 -- 172.21.15.12:0/723589893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12c406eb60 msgr2=0x7f12c412cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12c406eb60 0x7f12c412cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 monclient: found mon.noname-f 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c96e2700 1 -- 172.21.15.12:0/723589893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40094f0 con 0x7f12c40631f0 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.323+0000 7f12c9ee3700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 0x7f12c4099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12c96e2700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f12c000b810 tx=0x7f12c0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12c000f990 con 0x7f12c40631f0 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: dump: 2023-09-29T23:13:41.353 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.354 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 5 monclient: authenticate success, global_id 4366 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c000fb50 con 0x7f12c40631f0 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 10 monclient: _renew_subs 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c404c960 con 0x7f12c40631f0 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12c000fcc0 con 0x7f12c40631f0 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12b97fa700 10 monclient: dump: 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.356 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.357 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 10 monclient: _renew_subs 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c41275d0 con 0x7f12c40631f0 2023-09-29T23:13:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.325+0000 7f120c59a700 1 -- 172.21.15.12:0/1127731116 shutdown_connections 2023-09-29T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.324+0000 7f12bb7fe700 4 set_mon_vals no callback set 2023-09-29T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.325+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f12c000f510 con 0x7f12c40631f0 2023-09-29T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.325+0000 7f12b97fa700 1 --2- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 0x7f12a8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.325+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f12c0046e60 con 0x7f12c40631f0 2023-09-29T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.326+0000 7f12c8ee1700 1 --2- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 0x7f12a8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.326+0000 7f12c8ee1700 10 monclient: get_auth_request con 0x7f12a8036850 auth_method 0 2023-09-29T23:13:41.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.326+0000 7f12c8ee1700 1 --2- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 0x7f12a8038d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fd0 tx=0x7f12b4017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.329+0000 7f120c59a700 1 -- 172.21.15.12:0/1127731116 wait complete. 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.329+0000 7f120c59a700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.329+0000 7f120c59a700 10 monclient: monmap: 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.329+0000 7f120c59a700 1 Processor -- start 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 -- start start 2023-09-29T23:13:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient: init 2023-09-29T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f12cb946700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f12cb946700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f12cb946700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c406be80 con 0x7f12c40631f0 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient: _add_conns ranks=[5,6,0,2,8,3,4,1,7] 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f12040698a0 0x7f120413dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-h con 0x7f12040698a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-d con 0x7f120406f050 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120411d9b0 0x7f120411aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): picked mon.noname-a con 0x7f120411d9b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120409ef60 con 0x7f120411d9b0 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f12041350a0 con 0x7f12040698a0 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1204134dc0 con 0x7f120406f050 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120c59a700 10 monclient(hunting): authenticate will time out at 1130.690655s 2023-09-29T23:13:41.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56294/0 (socket says 172.21.15.12:56294) 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 1 -- 172.21.15.12:0/3599332666 learned_addr learned my addr 172.21.15.12:0/3599332666 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): get_auth_request con 0x7f120406f050 auth_method 0 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120ab37700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120411d9b0 0x7f120411aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120ab37700 10 monclient(hunting): get_auth_request con 0x7f120411d9b0 auth_method 0 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120ab37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120ab37700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f120ab37700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.330+0000 7f1209b35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120ab37700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 -- 172.21.15.12:0/3599332666 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f12040698a0 msgr2=0x7f120413dbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f12040698a0 0x7f120413dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 -- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120411d9b0 msgr2=0x7f120411aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120411d9b0 0x7f120411aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 monclient: found mon.noname-d 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 -- 172.21.15.12:0/3599332666 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc009a20 con 0x7f120406f050 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1209b35700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f11f400b840 tx=0x7f11f4009220 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f11f400f990 con 0x7f120406f050 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: dump: 2023-09-29T23:13:41.373 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.374 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 5 monclient: authenticate success, global_id 4344 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f400fb50 con 0x7f120406f050 2023-09-29T23:13:41.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f11f400fcc0 con 0x7f120406f050 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f1208b33700 4 set_mon_vals no callback set 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f11fa7fc700 10 monclient: dump: 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.377 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 10 monclient: _renew_subs 2023-09-29T23:13:41.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120404c960 con 0x7f120406f050 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 10 monclient: _renew_subs 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.331+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120411c650 con 0x7f120406f050 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.332+0000 7f120c59a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.332+0000 7f120c59a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.332+0000 7f120c59a700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.332+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204070bc0 con 0x7f120406f050 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.333+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f11f400f510 con 0x7f120406f050 2023-09-29T23:13:41.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.333+0000 7f11fa7fc700 1 --2- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 0x7f11e8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.333+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f11f4046ed0 con 0x7f120406f050 2023-09-29T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.337+0000 7f120a336700 1 --2- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 0x7f11e8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.337+0000 7f120a336700 10 monclient: get_auth_request con 0x7f11e8036850 auth_method 0 2023-09-29T23:13:41.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.338+0000 7f120a336700 1 --2- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 0x7f11e8038d00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ba0 tx=0x7f1200009220 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.340+0000 7fa67004d700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.340+0000 7fa67004d700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.340+0000 7fa67004d700 10 monclient: monmap: 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.341644+0000 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.341644+0000 2023-09-29T23:13:41.381 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.382 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.341+0000 7fa67004d700 1 Processor -- start 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.341+0000 7fa67004d700 1 -- start start 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.341+0000 7fa67004d700 10 monclient: init 2023-09-29T23:13:41.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient: _add_conns ranks=[10,7,2,3,1,4,9,8,6,0,5] 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa66812f660 0x7fa66812fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-f con 0x7fa66812f660 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 0x7fa66806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-i con 0x7fa66806eb60 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:41.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 0x7fa66809da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12c000f7c0 con 0x7f12c40631f0 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7f12b97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-b con 0x7fa66809d5a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66806bcf0 con 0x7fa66812f660 2023-09-29T23:13:41.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7f12b97fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66806d090 con 0x7fa66809d5a0 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa66812fa30 con 0x7fa66806eb60 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa67004d700 10 monclient(hunting): authenticate will time out at 1130.702555s 2023-09-29T23:13:41.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa66812f660 0x7fa66812fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa66812f660 0x7fa66812fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:60346/0 (socket says 172.21.15.12:60346) 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 1 -- 172.21.15.12:0/3505991885 learned_addr learned my addr 172.21.15.12:0/3505991885 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): get_auth_request con 0x7fa66812f660 auth_method 0 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66dde9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7fa66e5ea700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 0x7fa66809da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.342+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11f400fe40 con 0x7f120406f050 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7f11fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7f11fa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): get_auth_request con 0x7fa66809d5a0 auth_method 0 2023-09-29T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 0x7fa66806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 10 monclient(hunting): get_auth_request con 0x7fa66806eb60 auth_method 0 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 -- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 msgr2=0x7fa66806ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 0x7fa66806ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 -- 172.21.15.12:0/3505991885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 msgr2=0x7fa66809da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 0x7fa66809da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 monclient: found mon.noname-f 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 -- 172.21.15.12:0/3505991885 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668068ed0 con 0x7fa66812f660 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66e5ea700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 0x7fa66809da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66d5e8700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 0x7fa66806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:41.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66812f660 0x7fa66812fca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa66400ab50 tx=0x7fa66400a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b5332d3f9119826 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 1 -- 172.21.15.12:0/3505991885 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa66400cdf0 con 0x7fa66812f660 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: dump: 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.395 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 1 -- 172.21.15.12:0/3505991885 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa664010420 con 0x7fa66812f660 2023-09-29T23:13:41.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 1 -- 172.21.15.12:0/3505991885 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa664010a10 con 0x7fa66812f660 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.343+0000 7fa66cde7700 10 monclient: dump: 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.397 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.398 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 5 monclient: authenticate success, global_id 4375 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 4 set_mon_vals no callback set 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 10 monclient: shutdown 2023-09-29T23:13:41.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 1 -- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66812f660 msgr2=0x7fa66812fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7fa67004d700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66812f660 0x7fa66812fca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa66400ab50 tx=0x7fa66400a3f0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7f5216958700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7f5216958700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.346+0000 7f5216958700 10 monclient: monmap: 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.347751+0000 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.347751+0000 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.402 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 -- 172.21.15.12:0/3505991885 shutdown_connections 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66809d5a0 0x7fa66809da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa66812f660 0x7fa66812fca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 --2- 172.21.15.12:0/3505991885 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa66806eb60 0x7fa66806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 -- 172.21.15.12:0/3505991885 >> 172.21.15.12:0/3505991885 conn(0x7fa66806a370 msgr2=0x7fa6680a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 -- 172.21.15.12:0/3505991885 shutdown_connections 2023-09-29T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 1 -- 172.21.15.12:0/3505991885 wait complete. 2023-09-29T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7f5216958700 1 Processor -- start 2023-09-29T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7f5216958700 1 -- start start 2023-09-29T23:13:41.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7f5216958700 10 monclient: init 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.347+0000 7fa67004d700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7fa67004d700 10 monclient: monmap: 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.405 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.406 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient: _add_conns ranks=[3,2,6,9,10,5,1,4,7,0,8] 2023-09-29T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f521012d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): picked mon.noname-g con 0x7f521006ef50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:41.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f521012d000 0x7f521006f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7fa67004d700 1 Processor -- start 2023-09-29T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7fa67004d700 1 -- start start 2023-09-29T23:13:41.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7fa67004d700 10 monclient: init 2023-09-29T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): picked mon.noname-b con 0x7f521012d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): picked mon.noname-d con 0x7f52101300b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521006d280 con 0x7f521012d000 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521006d460 con 0x7f521006ef50 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f521006da20 con 0x7f52101300b0 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5216958700 10 monclient(hunting): authenticate will time out at 1130.708514s 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f521012d000 0x7f521006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56306/0 (socket says 172.21.15.12:56306) 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 1 -- 172.21.15.12:0/2955756874 learned_addr learned my addr 172.21.15.12:0/2955756874 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): get_auth_request con 0x7f52101300b0 auth_method 0 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): get_auth_request con 0x7f521012d000 auth_method 0 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.348+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 -- 172.21.15.12:0/2955756874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 msgr2=0x7f521012d630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f521012d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 -- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f521012d000 msgr2=0x7f521006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f521012d000 0x7f521006f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 monclient: found mon.noname-d 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 -- 172.21.15.12:0/2955756874 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210068ed0 con 0x7f52101300b0 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5214ef5700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5208014af0 tx=0x7f52080143c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=2451a3d8c671cea2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 1 -- 172.21.15.12:0/2955756874 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5208013f80 con 0x7f52101300b0 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: dump: 2023-09-29T23:13:41.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 1 -- 172.21.15.12:0/2955756874 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208014100 con 0x7f52101300b0 2023-09-29T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 1 -- 172.21.15.12:0/2955756874 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5208011550 con 0x7f52101300b0 2023-09-29T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient: _add_conns ranks=[2,8,4,7,6,3,5,1,0] 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66806eb60 0x7fa66813e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-b con 0x7fa66806eb60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa66809d5a0 0x7fa668099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-e con 0x7fa66809d5a0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): picked mon.noname-c con 0x7fa66812f660 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66806c5b0 con 0x7fa66806eb60 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa66809bc30 con 0x7fa66812f660 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa66806c170 con 0x7fa66809d5a0 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa67004d700 10 monclient(hunting): authenticate will time out at 1130.709504s 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37890/0 (socket says 172.21.15.12:37890) 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 1 -- 172.21.15.12:0/3413123810 learned_addr learned my addr 172.21.15.12:0/3413123810 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): get_auth_request con 0x7fa66812f660 auth_method 0 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66806eb60 0x7fa66813e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66dde9700 10 monclient(hunting): get_auth_request con 0x7fa66806eb60 auth_method 0 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66dde9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66dde9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66dde9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 -- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa66809d5a0 msgr2=0x7fa668099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa66809d5a0 0x7fa668099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 -- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66806eb60 msgr2=0x7fa66813e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66806eb60 0x7fa66813e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 monclient: found mon.noname-c 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 -- 172.21.15.12:0/3413123810 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660009a20 con 0x7fa66812f660 2023-09-29T23:13:41.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7fa66e5ea700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa66000e630 tx=0x7fa660009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f520effd700 10 monclient: dump: 2023-09-29T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.423 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5216958700 5 monclient: authenticate success, global_id 4353 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5216958700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.349+0000 7f5216958700 4 set_mon_vals no callback set 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 10 monclient: shutdown 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 -- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 msgr2=0x7f52101304e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5208014af0 tx=0x7f52080143c0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 -- 172.21.15.12:0/2955756874 shutdown_connections 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f52101304e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f521012d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 --2- 172.21.15.12:0/2955756874 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f521012d000 0x7f521006f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 -- 172.21.15.12:0/2955756874 >> 172.21.15.12:0/2955756874 conn(0x7f52100a3330 msgr2=0x7f52100a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 -- 172.21.15.12:0/2955756874 shutdown_connections 2023-09-29T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.350+0000 7f5216958700 1 -- 172.21.15.12:0/2955756874 wait complete. 2023-09-29T23:13:41.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa660012ac0 con 0x7fa66812f660 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: dump: 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.426 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660012c40 con 0x7fa66812f660 2023-09-29T23:13:41.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa660012db0 con 0x7fa66812f660 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65ffff700 4 set_mon_vals no callback set 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa65dffb700 10 monclient: dump: 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.428 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 5 monclient: authenticate success, global_id 4360 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 10 monclient: _renew_subs 2023-09-29T23:13:41.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66400a690 con 0x7fa66812f660 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 10 monclient: _renew_subs 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66804c960 con 0x7fa66812f660 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7f5216958700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7f5216958700 10 monclient: monmap: 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.351+0000 7f5216958700 1 Processor -- start 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 -- start start 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient: init 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa660003020 con 0x7fa66812f660 2023-09-29T23:13:41.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7fa65dffb700 1 --2- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 0x7fa64c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa660016070 con 0x7fa66812f660 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient: _add_conns ranks=[3,5,6,2,8,7,0,4,1] 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f5210098d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): picked mon.noname-g con 0x7f521006ef50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:41.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f521012d000 0x7f5210099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): picked mon.noname-h con 0x7f521012d000 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): picked mon.noname-d con 0x7f52101300b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521009f790 con 0x7f521006ef50 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f521009cba0 con 0x7f521012d000 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f521009c8c0 con 0x7f52101300b0 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.352+0000 7f5216958700 10 monclient(hunting): authenticate will time out at 1130.712987s 2023-09-29T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa67004d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa67004d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa67004d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668070bc0 con 0x7fa66812f660 2023-09-29T23:13:41.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 0x7fa64c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56316/0 (socket says 172.21.15.12:56316) 2023-09-29T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 -- 172.21.15.12:0/3382978878 learned_addr learned my addr 172.21.15.12:0/3382978878 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): get_auth_request con 0x7f52101300b0 auth_method 0 2023-09-29T23:13:41.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f521012d000 0x7f5210099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): get_auth_request con 0x7f521012d000 auth_method 0 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f5210098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 10 monclient(hunting): get_auth_request con 0x7f521006ef50 auth_method 0 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 -- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f521012d000 msgr2=0x7f5210099250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f521012d000 0x7f5210099250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.438 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 -- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 msgr2=0x7f5210098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f5210098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 monclient: found mon.noname-d 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 -- 172.21.15.12:0/3382978878 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208014600 con 0x7f52101300b0 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f520ffff700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f5210098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.353+0000 7f5214ef5700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5208013660 tx=0x7f520801f000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5208013f00 con 0x7f52101300b0 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: dump: 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.440 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 5 monclient: authenticate success, global_id 4362 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208026890 con 0x7f52101300b0 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5208026e80 con 0x7f52101300b0 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 10 monclient: _renew_subs 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521004c960 con 0x7f52101300b0 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f520dffb700 4 set_mon_vals no callback set 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f51f7fff700 10 monclient: dump: 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.442 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 10 monclient: _renew_subs 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.354+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521009d2a0 con 0x7f52101300b0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.355+0000 7f5216958700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.355+0000 7f5216958700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.355+0000 7f5216958700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.355+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521009c240 con 0x7f52101300b0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.356+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5208023070 con 0x7f52101300b0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.356+0000 7f51f7fff700 1 --2- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 0x7f51fc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.356+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f52080593b0 con 0x7f52101300b0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.357+0000 7f520ffff700 1 --2- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 0x7f51fc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.357+0000 7f520ffff700 10 monclient: get_auth_request con 0x7f51fc036880 auth_method 0 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.357+0000 7f520ffff700 1 --2- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 0x7f51fc038d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5200006fd0 tx=0x7f5200008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.361+0000 7fa66dde9700 10 monclient: get_auth_request con 0x7fa64c036850 auth_method 0 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.362+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa660007d80 con 0x7fa66812f660 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.362+0000 7fa65dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.362+0000 7fa65dffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.362+0000 7fa66dde9700 1 --2- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 0x7fa64c038d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa66400a660 tx=0x7fa664017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.364+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f520801a310 con 0x7f52101300b0 2023-09-29T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.364+0000 7f51f7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.364+0000 7f51f7fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f99905f1700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f99905f1700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f99905f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9988066790 con 0x7f998812f660 2023-09-29T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f997e7fc700 1 -- 172.21.15.12:0/3854800857 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9980035090 con 0x7f998812f660 2023-09-29T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f997e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-29T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.497+0000 7f997e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.498 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 msgr2=0x7f996c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 0x7f996c038d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d80 tx=0x7f9984008040 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 10 monclient: shutdown 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 msgr2=0x7f998809b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.500+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9980002d30 tx=0x7f9980016040 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 shutdown_connections 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> v2:172.21.15.153:6812/83257 conn(0x7f996c036880 0x7f996c038d30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f998812f660 0x7f998809b1f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f998806eb60 0x7f998812bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 --2- 172.21.15.12:0/3854800857 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f998809d5a0 0x7f998809acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.501+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 >> 172.21.15.12:0/3854800857 conn(0x7f998806a370 msgr2=0x7f998806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.502+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 shutdown_connections 2023-09-29T23:13:41.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.502+0000 7f99905f1700 1 -- 172.21.15.12:0/3854800857 wait complete. 2023-09-29T23:13:41.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-09-29T23:13:41.518 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.569+0000 7fa67004d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.569+0000 7fa67004d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.569+0000 7fa67004d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.569+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa66812ce90 con 0x7fa66812f660 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.570+0000 7fa65dffb700 1 -- 172.21.15.12:0/3413123810 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa660007340 con 0x7fa66812f660 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.570+0000 7fa65dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.570+0000 7fa65dffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.570 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 msgr2=0x7fa64c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 0x7fa64c038d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa66400a660 tx=0x7fa664017000 comp rx=0 tx=0).stop 2023-09-29T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 10 monclient: shutdown 2023-09-29T23:13:41.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 msgr2=0x7fa668099fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa66000e630 tx=0x7fa660009780 comp rx=0 tx=0).stop 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 shutdown_connections 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa66812f660 0x7fa668099fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> v2:172.21.15.153:6812/83257 conn(0x7fa64c036850 0x7fa64c038d00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa66806eb60 0x7fa66813e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 --2- 172.21.15.12:0/3413123810 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa66809d5a0 0x7fa668099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 >> 172.21.15.12:0/3413123810 conn(0x7fa66806a370 msgr2=0x7fa66806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 shutdown_connections 2023-09-29T23:13:41.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.572+0000 7fa67004d700 1 -- 172.21.15.12:0/3413123810 wait complete. 2023-09-29T23:13:41.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2023-09-29T23:13:41.585 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.584+0000 7fb1eafb3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.584+0000 7fb1eafb3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.584+0000 7fb1eafb3700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.584+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb1c4005620 con 0x7fb1e406eff0 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.585+0000 7fb1e0ff9700 1 -- 172.21.15.12:0/2154138077 <== mon.5 v2:172.21.15.153:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb1d8017070 con 0x7fb1e406eff0 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.585+0000 7fb1e0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.585+0000 7fb1e0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.586 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 msgr2=0x7fb1c8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 0x7fb1c8038d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00097b0 tx=0x7fb1d0017000 comp rx=0 tx=0).stop 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 10 monclient: shutdown 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 msgr2=0x7fb1e409dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 0x7fb1e409dfa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c560 tx=0x7fb1d8007d10 comp rx=0 tx=0).stop 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 shutdown_connections 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> v2:172.21.15.153:6812/83257 conn(0x7fb1c8036850 0x7fb1c8038d00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb1e4136370 0x7fb1e4136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fb1e406eff0 0x7fb1e409dfa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 --2- 172.21.15.12:0/2154138077 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fb1e406f3c0 0x7fb1e4136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.588+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 >> 172.21.15.12:0/2154138077 conn(0x7fb1e40a0ea0 msgr2=0x7fb1e40969e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.589+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 shutdown_connections 2023-09-29T23:13:41.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.589+0000 7fb1eafb3700 1 -- 172.21.15.12:0/2154138077 wait complete. 2023-09-29T23:13:41.604 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2023-09-29T23:13:41.604 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f120c59a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f120c59a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f120c59a700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f120409f610 con 0x7f120406f050 2023-09-29T23:13:41.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f11fa7fc700 1 -- 172.21.15.12:0/3599332666 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11f4018300 con 0x7f120406f050 2023-09-29T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f11fa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-29T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.649+0000 7f11fa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.650 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 msgr2=0x7f11e8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 0x7f11e8038d00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ba0 tx=0x7f1200009220 comp rx=0 tx=0).stop 2023-09-29T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 10 monclient: shutdown 2023-09-29T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 msgr2=0x7f120411aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f11f400b840 tx=0x7f11f4009220 comp rx=0 tx=0).stop 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 shutdown_connections 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> v2:172.21.15.153:6812/83257 conn(0x7f11e8036850 0x7f11e8038d00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f120411d9b0 0x7f120411aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f12040698a0 0x7f120413dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 --2- 172.21.15.12:0/3599332666 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f120406f050 0x7f120411aab0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.651+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 >> 172.21.15.12:0/3599332666 conn(0x7f12040a3330 msgr2=0x7f12040a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.652+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 shutdown_connections 2023-09-29T23:13:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.652+0000 7f120c59a700 1 -- 172.21.15.12:0/3599332666 wait complete. 2023-09-29T23:13:41.664 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2023-09-29T23:13:41.664 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f5216958700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f5216958700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f5216958700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5210070bc0 con 0x7f52101300b0 2023-09-29T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f51f7fff700 1 -- 172.21.15.12:0/3382978878 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5208021070 con 0x7f52101300b0 2023-09-29T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f51f7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-29T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.667+0000 7f51f7fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.668 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.669+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 msgr2=0x7f51fc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.669+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 0x7f51fc038d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5200006fd0 tx=0x7f5200008040 comp rx=0 tx=0).stop 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.669+0000 7f5216958700 10 monclient: shutdown 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 msgr2=0x7f5210099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5208013660 tx=0x7f520801f000 comp rx=0 tx=0).stop 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 shutdown_connections 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f52101300b0 0x7f5210099790 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> v2:172.21.15.153:6812/83257 conn(0x7f51fc036880 0x7f51fc038d30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f521006ef50 0x7f5210098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 --2- 172.21.15.12:0/3382978878 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f521012d000 0x7f5210099250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 >> 172.21.15.12:0/3382978878 conn(0x7f52100a3330 msgr2=0x7f521009dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 shutdown_connections 2023-09-29T23:13:41.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.670+0000 7f5216958700 1 -- 172.21.15.12:0/3382978878 wait complete. 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.671+0000 7f12cb946700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.671+0000 7f12cb946700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.671+0000 7f12cb946700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.671+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f12c4070bc0 con 0x7f12c40631f0 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.672+0000 7f12b97fa700 1 -- 172.21.15.12:0/723589893 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f12c0018300 con 0x7f12c40631f0 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.672+0000 7f12b97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.672+0000 7f12b97fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:41.673 INFO:teuthology.orchestra.run.smithi012.stdout:47244640259 2023-09-29T23:13:41.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 msgr2=0x7f12a8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 0x7f12a8038d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fd0 tx=0x7f12b4017000 comp rx=0 tx=0).stop 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 10 monclient: shutdown 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 msgr2=0x7f12c41425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f12c000b810 tx=0x7f12c0009250 comp rx=0 tx=0).stop 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 shutdown_connections 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> v2:172.21.15.153:6812/83257 conn(0x7f12a8036850 0x7f12a8038d00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12c412f660 0x7f12c4099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12c40631f0 0x7f12c41425e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 --2- 172.21.15.12:0/723589893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12c406eb60 0x7f12c412cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 >> 172.21.15.12:0/723589893 conn(0x7f12c406a370 msgr2=0x7f12c406da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 shutdown_connections 2023-09-29T23:13:41.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.674+0000 7f12cb946700 1 -- 172.21.15.12:0/723589893 wait complete. 2023-09-29T23:13:41.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-09-29T23:13:41.683 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2023-09-29T23:13:41.687 DEBUG:teuthology.parallel:result is None 2023-09-29T23:13:41.687 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-29T23:13:41.687 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.845+0000 7f47633a4700 10 monclient: get_monmap_and_config 2023-09-29T23:13:41.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.845+0000 7f47633a4700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.845+0000 7f47633a4700 10 monclient: monmap: 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:41.846150+0000 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:41.846150+0000 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.846 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.845+0000 7f47633a4700 1 Processor -- start 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 -- start start 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient: init 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient: _add_conns ranks=[9,8,6,7,1,2,10,5,4,0,3] 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 0x7f475c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-a con 0x7f475c0a1f90 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:41.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-e con 0x7f475c0a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-d con 0x7f475c133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475c09e350 con 0x7f475c0a1f90 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f475c09ef70 con 0x7f475c133740 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f475c09da70 con 0x7f475c0a2360 2023-09-29T23:13:41.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f47633a4700 10 monclient(hunting): authenticate will time out at 1131.206567s 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 0x7f475c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:56326/0 (socket says 172.21.15.12:56326) 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55392/0 (socket says 172.21.15.12:55392) 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 1 -- 172.21.15.12:0/1700525477 learned_addr learned my addr 172.21.15.12:0/1700525477 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): get_auth_request con 0x7f475c0a1f90 auth_method 0 2023-09-29T23:13:41.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): get_auth_request con 0x7f475c133740 auth_method 0 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): get_auth_request con 0x7f475c0a2360 auth_method 0 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761941700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.846+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 -- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 msgr2=0x7f475c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 --2- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 -- 172.21.15.12:0/1700525477 >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 msgr2=0x7f475c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 --2- 172.21.15.12:0/1700525477 >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 0x7f475c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 monclient: found mon.noname-d 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 -- 172.21.15.12:0/1700525477 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c068ed0 con 0x7f475c133740 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761140700 1 --2- 172.21.15.12:0/1700525477 >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 0x7f475c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4761941700 1 --2- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4754009f50 tx=0x7f4754009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7315300c9bb80767 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 1 -- 172.21.15.12:0/1700525477 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f475400e070 con 0x7f475c133740 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: dump: 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 1 -- 172.21.15.12:0/1700525477 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475400bd90 con 0x7f475c133740 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 5 monclient: authenticate success, global_id 4371 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 1 -- 172.21.15.12:0/1700525477 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f47540092e0 con 0x7f475c133740 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 4 set_mon_vals no callback set 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f4753fff700 10 monclient: dump: 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.865 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.866 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 10 monclient: shutdown 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 -- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 msgr2=0x7f475c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 --2- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4754009f50 tx=0x7f4754009840 comp rx=0 tx=0).stop 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 -- 172.21.15.12:0/1700525477 shutdown_connections 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 --2- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f475c133740 0x7f475c133be0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 --2- 172.21.15.12:0/1700525477 >> v2:172.21.15.12:3300/0 conn(0x7f475c0a1f90 0x7f475c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 --2- 172.21.15.12:0/1700525477 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.847+0000 7f47633a4700 1 -- 172.21.15.12:0/1700525477 >> 172.21.15.12:0/1700525477 conn(0x7f475c0a3b50 msgr2=0x7f475c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.848+0000 7f47633a4700 1 -- 172.21.15.12:0/1700525477 shutdown_connections 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.848+0000 7f47633a4700 1 -- 172.21.15.12:0/1700525477 wait complete. 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.848+0000 7f47633a4700 10 monclient: build_initial_monmap 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.848+0000 7f47633a4700 10 monclient: monmap: 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:41.868 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.848+0000 7f47633a4700 1 Processor -- start 2023-09-29T23:13:41.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 -- start start 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient: init 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient: _add_conns ranks=[0,8,5,4,7,1,3,2,6] 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-a con 0x7f475c0a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c09c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-e con 0x7f475c0a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f475c133740 0x7f475c09cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): picked mon.noname-h con 0x7f475c133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c131b10 con 0x7f475c0a1f90 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f475c12ed30 con 0x7f475c133740 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f475c09ffc0 con 0x7f475c0a2360 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f47633a4700 10 monclient(hunting): authenticate will time out at 1131.209657s 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53020/0 (socket says 172.21.15.12:53020) 2023-09-29T23:13:41.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c09c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 1 -- 172.21.15.12:0/1417790856 learned_addr learned my addr 172.21.15.12:0/1417790856 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): get_auth_request con 0x7f475c0a1f90 auth_method 0 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): get_auth_request con 0x7f475c0a2360 auth_method 0 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f476093f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761941700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f475c133740 0x7f475c09cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761941700 10 monclient(hunting): get_auth_request con 0x7f475c133740 auth_method 0 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761941700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761941700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761941700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:41.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.849+0000 7f4761140700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 -- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 msgr2=0x7f475c09c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c09c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 -- 172.21.15.12:0/1417790856 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f475c133740 msgr2=0x7f475c09cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f475c133740 0x7f475c09cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 10 monclient: _finish_hunting 0 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 monclient: found mon.noname-a 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 -- 172.21.15.12:0/1417790856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4758009420 con 0x7f475c0a1f90 2023-09-29T23:13:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f4761140700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f475800ec80 tx=0x7f47580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f475800ccd0 con 0x7f475c0a1f90 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: dump: 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.876 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 5 monclient: authenticate success, global_id 4464 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475800ce30 con 0x7f475c0a1f90 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 10 monclient: _renew_subs 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4754009ae0 con 0x7f475c0a1f90 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:41.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f47580138f0 con 0x7f475c0a1f90 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47537fe700 4 set_mon_vals no callback set 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47517fa700 10 monclient: dump: 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:41.878 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:41.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 10 monclient: _renew_subs 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.850+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475c04c960 con 0x7f475c0a1f90 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4758013470 con 0x7f475c0a1f90 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47633a4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47633a4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47633a4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475c070bc0 con 0x7f475c0a1f90 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47517fa700 1 --2- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 0x7f4740038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4758047950 con 0x7f475c0a1f90 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f476093f700 1 --2- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 0x7f4740038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:41.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.851+0000 7f476093f700 10 monclient: get_auth_request con 0x7f4740036850 auth_method 0 2023-09-29T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.852+0000 7f476093f700 1 --2- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 0x7f4740038d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f475c12eb60 tx=0x7f474c0091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.856+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4758013720 con 0x7f475c0a1f90 2023-09-29T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.856+0000 7f47517fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:41.856+0000 7f47517fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:42.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.043+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f475c069190 con 0x7f4740036850 2023-09-29T23:13:42.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.045+0000 7f47517fa700 1 -- 172.21.15.12:0/1417790856 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38708 (secure 0 0 0) 0x7f474c011a20 con 0x7f4740036850 2023-09-29T23:13:42.046 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 msgr2=0x7f4740038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 0x7f4740038d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f475c12eb60 tx=0x7f474c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 10 monclient: shutdown 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 msgr2=0x7f475c09c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f475800ec80 tx=0x7f47580091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 shutdown_connections 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> v2:172.21.15.153:6812/83257 conn(0x7f4740036850 0x7f4740038d00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f475c133740 0x7f475c09cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f475c0a1f90 0x7f475c09c480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 --2- 172.21.15.12:0/1417790856 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f475c0a2360 0x7f475c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 >> 172.21.15.12:0/1417790856 conn(0x7f475c0a3b50 msgr2=0x7f475c135f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 shutdown_connections 2023-09-29T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.047+0000 7f47633a4700 1 -- 172.21.15.12:0/1417790856 wait complete. 2023-09-29T23:13:42.049 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:13:42.059 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T23:13:40.724185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4780436,"kb_used_data":557418944,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557415844,"statfs":{"total":575688990720,"available":570793824256,"internally_reserved":3174400,"allocated":570796998656,"data_stored":570796998656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422282,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.597041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.129207+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:38.129207+0000","last_peered":"2023-09-29T23:13:38.129207+0000","last_clean":"2023-09-29T23:13:38.129207+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:38.129207+0000","last_undegraded":"2023-09-29T23:13:38.129207+0000","last_fullsized":"2023-09-29T23:13:38.129207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:35.802738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.130672+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:38.130672+0000","last_peered":"2023-09-29T23:13:38.130672+0000","last_clean":"2023-09-29T23:13:38.130672+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:38.130672+0000","last_undegraded":"2023-09-29T23:13:38.130672+0000","last_fullsized":"2023-09-29T23:13:38.130672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:01.042284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127425+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:38.127425+0000","last_peered":"2023-09-29T23:13:38.127425+0000","last_clean":"2023-09-29T23:13:38.127425+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:38.127425+0000","last_undegraded":"2023-09-29T23:13:38.127425+0000","last_fullsized":"2023-09-29T23:13:38.127425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:56.316625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.128695+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:38.128695+0000","last_peered":"2023-09-29T23:13:38.128695+0000","last_clean":"2023-09-29T23:13:38.128695+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:38.128695+0000","last_undegraded":"2023-09-29T23:13:38.128695+0000","last_fullsized":"2023-09-29T23:13:38.128695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:59:39.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126113+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:38.126113+0000","last_peered":"2023-09-29T23:13:38.126113+0000","last_clean":"2023-09-29T23:13:38.126113+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:38.126113+0000","last_undegraded":"2023-09-29T23:13:38.126113+0000","last_fullsized":"2023-09-29T23:13:38.126113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:24.806629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":800320,"kb_used_data":92902152,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92899060,"statfs":{"total":95948165120,"available":95128637440,"internally_reserved":3166208,"allocated":95131803648,"data_stored":95131803648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118954,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795824,"kb_used_data":92903564,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903556,"statfs":{"total":95948165120,"available":95133241344,"internally_reserved":8192,"allocated":95133249536,"data_stored":95133249536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48915,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796988,"kb_used_data":92902392,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902392,"statfs":{"total":95948165120,"available":95132049408,"internally_reserved":0,"allocated":95132049408,"data_stored":95132049408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118325,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43596,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48877,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43615,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]}],"pool_statfs":[]}} 2023-09-29T23:13:42.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 10 monclient: get_monmap_and_config 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 10 monclient: build_initial_monmap 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 10 monclient: monmap: 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:42.180978+0000 2023-09-29T23:13:42.180 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:42.180978+0000 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:42.181 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:42.182 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 1 Processor -- start 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 1 -- start start 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.180+0000 7f9aceb62700 10 monclient: init 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient: _add_conns ranks=[8,2,5,6,3,1,7,10,0,9,4] 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 0x7f9ac809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-e con 0x7f9ac80a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:42.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-b con 0x7f9ac80a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9ac8133740 0x7f9ac8133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-h con 0x7f9ac8133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac809e350 con 0x7f9ac80a2360 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac809ef70 con 0x7f9ac8133740 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac809da70 con 0x7f9ac80a1f90 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9aceb62700 10 monclient(hunting): authenticate will time out at 1131.541362s 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34794/0 (socket says 172.21.15.12:34794) 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 -- 172.21.15.12:0/4182790590 learned_addr learned my addr 172.21.15.12:0/4182790590 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): get_auth_request con 0x7f9ac80a2360 auth_method 0 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 0x7f9ac809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): get_auth_request con 0x7f9ac80a1f90 auth_method 0 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acd0ff700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9ac8133740 0x7f9ac8133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acd0ff700 10 monclient(hunting): get_auth_request con 0x7f9ac8133740 auth_method 0 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acd0ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acd0ff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acd0ff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 -- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 msgr2=0x7f9ac809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 0x7f9ac809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 -- 172.21.15.12:0/4182790590 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9ac8133740 msgr2=0x7f9ac8133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9ac8133740 0x7f9ac8133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 monclient: found mon.noname-b 2023-09-29T23:13:42.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9ac7fff700 1 -- 172.21.15.12:0/4182790590 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8068ed0 con 0x7f9ac80a2360 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.181+0000 7f9acc8fe700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 0x7f9ac809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009a10 tx=0x7f9abc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df3d0e71c60df86e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 1 -- 172.21.15.12:0/4182790590 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9abc00bca0 con 0x7f9ac80a2360 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: dump: 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.196 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.197 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: _finish_auth 0 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 1 -- 172.21.15.12:0/4182790590 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc00be20 con 0x7f9ac80a2360 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 5 monclient: authenticate success, global_id 4295 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 1 -- 172.21.15.12:0/4182790590 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9abc0118a0 con 0x7f9ac80a2360 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:42.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9ac77fe700 10 monclient: dump: 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.199 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 4 set_mon_vals no callback set 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 10 monclient: shutdown 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 -- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 msgr2=0x7f9ac80a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009a10 tx=0x7f9abc009250 comp rx=0 tx=0).stop 2023-09-29T23:13:42.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 -- 172.21.15.12:0/4182790590 shutdown_connections 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9ac8133740 0x7f9ac8133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9ac80a1f90 0x7f9ac809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 --2- 172.21.15.12:0/4182790590 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a2360 0x7f9ac80a27b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 -- 172.21.15.12:0/4182790590 >> 172.21.15.12:0/4182790590 conn(0x7f9ac80a3b50 msgr2=0x7f9ac80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 -- 172.21.15.12:0/4182790590 shutdown_connections 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.182+0000 7f9aceb62700 1 -- 172.21.15.12:0/4182790590 wait complete. 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.183+0000 7f9aceb62700 10 monclient: build_initial_monmap 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.183+0000 7f9aceb62700 10 monclient: monmap: 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:42.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:42.202 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.183+0000 7f9aceb62700 1 Processor -- start 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.183+0000 7f9aceb62700 1 -- start start 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.183+0000 7f9aceb62700 10 monclient: init 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient: _add_conns ranks=[2,6,3,8,5,0,7,4,1] 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-b con 0x7f9ac80a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 0x7f9ac809cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-d con 0x7f9ac80a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 0x7f9ac812d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): picked mon.noname-g con 0x7f9ac8133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8130d50 con 0x7f9ac80a1f90 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac812efb0 con 0x7f9ac8133740 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8130a70 con 0x7f9ac80a2360 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9aceb62700 10 monclient(hunting): authenticate will time out at 1131.544446s 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34804/0 (socket says 172.21.15.12:34804) 2023-09-29T23:13:42.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 -- 172.21.15.12:0/2404418445 learned_addr learned my addr 172.21.15.12:0/2404418445 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): get_auth_request con 0x7f9ac80a1f90 auth_method 0 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 0x7f9ac809cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): get_auth_request con 0x7f9ac80a2360 auth_method 0 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acd0ff700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 0x7f9ac812d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 -- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 msgr2=0x7f9ac809cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 0x7f9ac809cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 -- 172.21.15.12:0/2404418445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 msgr2=0x7f9ac812d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 0x7f9ac812d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:42.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 monclient: found mon.noname-b 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acc8fe700 1 -- 172.21.15.12:0/2404418445 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009420 con 0x7f9ac80a1f90 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acd0ff700 10 monclient: get_auth_request con 0x7f9ac8133740 auth_method 0 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 0x7f9ac809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.184+0000 7f9acd0ff700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 0x7f9ac812d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9acc8fe700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800ec80 tx=0x7f9ab80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9ab800cd70 con 0x7f9ac80a1f90 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:42.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: dump: 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.208 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: _finish_auth 0 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 5 monclient: authenticate success, global_id 4304 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800ced0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 10 monclient: _renew_subs 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc0094f0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9ab80138f0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac6ffd700 4 set_mon_vals no callback set 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 10 monclient: dump: 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.210 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.211 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 10 monclient: _renew_subs 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac804c960 con 0x7f9ac80a1f90 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.185+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9ab8013470 con 0x7f9ac80a1f90 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9ac4ff9700 1 --2- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 0x7f9ab4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9ab80479f0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9aceb62700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9aceb62700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9aceb62700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac8070bc0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 0x7f9ab4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9ac7fff700 10 monclient: get_auth_request con 0x7f9ab4036850 auth_method 0 2023-09-29T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.186+0000 7f9ac7fff700 1 --2- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 0x7f9ab4038d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806b460 tx=0x7f9abc017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.191+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ab801b3b0 con 0x7f9ac80a1f90 2023-09-29T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.191+0000 7f9ac4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.191+0000 7f9ac4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.380+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ac809e490 con 0x7f9ab4036850 2023-09-29T23:13:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.382+0000 7f9ac4ff9700 1 -- 172.21.15.12:0/2404418445 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38708 (secure 0 0 0) 0x7f9ac809e490 con 0x7f9ab4036850 2023-09-29T23:13:42.383 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 msgr2=0x7f9ab4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 0x7f9ab4038d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806b460 tx=0x7f9abc017000 comp rx=0 tx=0).stop 2023-09-29T23:13:42.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 10 monclient: shutdown 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 msgr2=0x7f9ac809c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800ec80 tx=0x7f9ab80091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 shutdown_connections 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> v2:172.21.15.153:6812/83257 conn(0x7f9ab4036850 0x7f9ab4038d00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ac8133740 0x7f9ac812d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac80a1f90 0x7f9ac809c6d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 --2- 172.21.15.12:0/2404418445 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9ac80a2360 0x7f9ac809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 >> 172.21.15.12:0/2404418445 conn(0x7f9ac80a3b50 msgr2=0x7f9ac806f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 shutdown_connections 2023-09-29T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.385+0000 7f9aceb62700 1 -- 172.21.15.12:0/2404418445 wait complete. 2023-09-29T23:13:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:13:42.397 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T23:13:40.724185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4780436,"kb_used_data":557418944,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557415844,"statfs":{"total":575688990720,"available":570793824256,"internally_reserved":3174400,"allocated":570796998656,"data_stored":570796998656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422282,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.597041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.129207+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:38.129207+0000","last_peered":"2023-09-29T23:13:38.129207+0000","last_clean":"2023-09-29T23:13:38.129207+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:38.129207+0000","last_undegraded":"2023-09-29T23:13:38.129207+0000","last_fullsized":"2023-09-29T23:13:38.129207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:35.802738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.130672+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:38.130672+0000","last_peered":"2023-09-29T23:13:38.130672+0000","last_clean":"2023-09-29T23:13:38.130672+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:38.130672+0000","last_undegraded":"2023-09-29T23:13:38.130672+0000","last_fullsized":"2023-09-29T23:13:38.130672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:01.042284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127425+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:38.127425+0000","last_peered":"2023-09-29T23:13:38.127425+0000","last_clean":"2023-09-29T23:13:38.127425+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:38.127425+0000","last_undegraded":"2023-09-29T23:13:38.127425+0000","last_fullsized":"2023-09-29T23:13:38.127425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:56.316625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.128695+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:38.128695+0000","last_peered":"2023-09-29T23:13:38.128695+0000","last_clean":"2023-09-29T23:13:38.128695+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:38.128695+0000","last_undegraded":"2023-09-29T23:13:38.128695+0000","last_fullsized":"2023-09-29T23:13:38.128695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:59:39.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126113+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:38.126113+0000","last_peered":"2023-09-29T23:13:38.126113+0000","last_clean":"2023-09-29T23:13:38.126113+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:38.126113+0000","last_undegraded":"2023-09-29T23:13:38.126113+0000","last_fullsized":"2023-09-29T23:13:38.126113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:24.806629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":800320,"kb_used_data":92902152,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92899060,"statfs":{"total":95948165120,"available":95128637440,"internally_reserved":3166208,"allocated":95131803648,"data_stored":95131803648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118954,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795824,"kb_used_data":92903564,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903556,"statfs":{"total":95948165120,"available":95133241344,"internally_reserved":8192,"allocated":95133249536,"data_stored":95133249536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48915,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796988,"kb_used_data":92902392,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902392,"statfs":{"total":95948165120,"available":95132049408,"internally_reserved":0,"allocated":95132049408,"data_stored":95132049408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118325,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43596,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48877,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795768,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903612,"statfs":{"total":95948165120,"available":95133298688,"internally_reserved":0,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43615,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]}],"pool_statfs":[]}} 2023-09-29T23:13:42.398 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-29T23:13:42.398 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-29T23:13:42.398 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-29T23:13:42.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.515+0000 7f93da60e700 10 monclient: get_monmap_and_config 2023-09-29T23:13:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.515+0000 7f93da60e700 10 monclient: build_initial_monmap 2023-09-29T23:13:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.515+0000 7f93da60e700 10 monclient: monmap: 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:42.516344+0000 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:42.516344+0000 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:42.517 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 Processor -- start 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 -- start start 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient: init 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient: _add_conns ranks=[2,9,1,0,10,5,4,3,6,8,7] 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-b con 0x7f93d40a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-a con 0x7f93d40a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f93d4133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4068ed0 con 0x7f93d4133740 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d409e350 con 0x7f93d40a2360 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d409ef70 con 0x7f93d40a1f90 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93da60e700 10 monclient(hunting): authenticate will time out at 1131.876833s 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34812/0 (socket says 172.21.15.12:34812) 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53028/0 (socket says 172.21.15.12:53028) 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 1 -- 172.21.15.12:0/2784237304 learned_addr learned my addr 172.21.15.12:0/2784237304 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 10 monclient(hunting): get_auth_request con 0x7f93d40a1f90 auth_method 0 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 10 monclient(hunting): get_auth_request con 0x7f93d40a2360 auth_method 0 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.516+0000 7f93d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 -- 172.21.15.12:0/2784237304 >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 msgr2=0x7f93d40a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 --2- 172.21.15.12:0/2784237304 >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 -- 172.21.15.12:0/2784237304 >> v1:172.21.15.153:6789/0 conn(0x7f93d4133740 legacy=0x7f93d4133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 monclient: found mon.noname-b 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 -- 172.21.15.12:0/2784237304 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d404c960 con 0x7f93d40a1f90 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d37fe700 1 --2- 172.21.15.12:0/2784237304 >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d3fff700 1 --2- 172.21.15.12:0/2784237304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f93c40099b0 tx=0x7f93c4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67c209a44934489 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 1 -- 172.21.15.12:0/2784237304 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93c4009ea0 con 0x7f93d40a1f90 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: dump: 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93da60e700 5 monclient: authenticate success, global_id 4313 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93da60e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 1 -- 172.21.15.12:0/2784237304 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c400bca0 con 0x7f93d40a1f90 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93da60e700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 1 -- 172.21.15.12:0/2784237304 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93c400be10 con 0x7f93d40a1f90 2023-09-29T23:13:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93da60e700 4 set_mon_vals no callback set 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.517+0000 7f93d2ffd700 10 monclient: dump: 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 10 monclient: shutdown 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 -- 172.21.15.12:0/2784237304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 msgr2=0x7f93d409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 --2- 172.21.15.12:0/2784237304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f93c40099b0 tx=0x7f93c4009250 comp rx=0 tx=0).stop 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 -- 172.21.15.12:0/2784237304 shutdown_connections 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 --2- 172.21.15.12:0/2784237304 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d409f5e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 --2- 172.21.15.12:0/2784237304 >> v2:172.21.15.12:3300/0 conn(0x7f93d40a2360 0x7f93d40a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 -- 172.21.15.12:0/2784237304 >> 172.21.15.12:0/2784237304 conn(0x7f93d40a3b50 msgr2=0x7f93d40a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 -- 172.21.15.12:0/2784237304 shutdown_connections 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 1 -- 172.21.15.12:0/2784237304 wait complete. 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 10 monclient: build_initial_monmap 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.518+0000 7f93da60e700 10 monclient: monmap: 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 Processor -- start 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 -- start start 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient: init 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient: _add_conns ranks=[2,8,5,4,3,1,0,6,7] 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-b con 0x7f93d40a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 0x7f93d41329b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-e con 0x7f93d40a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93d4133740 0x7f93d412d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): picked mon.noname-h con 0x7f93d4133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4130be0 con 0x7f93d40a1f90 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f93d409ffc0 con 0x7f93d4133740 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f93d412f5c0 con 0x7f93d40a2360 2023-09-29T23:13:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93da60e700 10 monclient(hunting): authenticate will time out at 1131.879751s 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 0x7f93d41329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:34826/0 (socket says 172.21.15.12:34826) 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 1 -- 172.21.15.12:0/4009595539 learned_addr learned my addr 172.21.15.12:0/4009595539 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): get_auth_request con 0x7f93d40a2360 auth_method 0 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): get_auth_request con 0x7f93d40a1f90 auth_method 0 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d8bab700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93d4133740 0x7f93d412d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d8bab700 10 monclient(hunting): get_auth_request con 0x7f93d4133740 auth_method 0 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.519+0000 7f93d8bab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d8bab700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d8bab700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d8bab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d8bab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d8bab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 -- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 msgr2=0x7f93d41329b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 0x7f93d41329b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 -- 172.21.15.12:0/4009595539 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93d4133740 msgr2=0x7f93d412d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93d4133740 0x7f93d412d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 10 monclient: _finish_hunting 0 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 monclient: found mon.noname-b 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 -- 172.21.15.12:0/4009595539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c8009420 con 0x7f93d40a1f90 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d37fe700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 0x7f93d41329b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d3fff700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f93c4000c00 tx=0x7f93c4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93c4011dd0 con 0x7f93d40a1f90 2023-09-29T23:13:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: dump: 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: _finish_auth 0 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c4020070 con 0x7f93d40a1f90 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 5 monclient: authenticate success, global_id 4322 2023-09-29T23:13:42.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 10 monclient: _renew_subs 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c40094f0 con 0x7f93d40a1f90 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93c401c8d0 con 0x7f93d40a1f90 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93bbfff700 10 monclient: dump: 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93d27fc700 4 set_mon_vals no callback set 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 10 monclient: _renew_subs 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.520+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d412fc80 con 0x7f93d40a1f90 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f93c4022030 con 0x7f93d40a1f90 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93da60e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93da60e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93da60e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d41310e0 con 0x7f93d40a1f90 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93bbfff700 1 --2- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 0x7f93c0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f93c4048cb0 con 0x7f93d40a1f90 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93d37fe700 1 --2- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 0x7f93c0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.521+0000 7f93d37fe700 10 monclient: get_auth_request con 0x7f93c0036850 auth_method 0 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.522+0000 7f93d37fe700 1 --2- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 0x7f93c0038d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93d4069720 tx=0x7f93c80091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.526+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93d41310e0 con 0x7f93d40a1f90 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.526+0000 7f93bbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:42.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.526+0000 7f93bbfff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:42.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.763+0000 7f93da60e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:42.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.763+0000 7f93da60e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:42.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.763+0000 7f93da60e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.763+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f93d4070bc0 con 0x7f93d40a1f90 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.764+0000 7f93bbfff700 1 -- 172.21.15.12:0/4009595539 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f93c401ad60 con 0x7f93d40a1f90 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.764+0000 7f93bbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.764+0000 7f93bbfff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:42.765 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:42.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 msgr2=0x7f93c0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 0x7f93c0038d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93d4069720 tx=0x7f93c80091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 10 monclient: shutdown 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 msgr2=0x7f93d4142370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f93c4000c00 tx=0x7f93c4016000 comp rx=0 tx=0).stop 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 shutdown_connections 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93d4133740 0x7f93d412d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> v2:172.21.15.153:6812/83257 conn(0x7f93c0036850 0x7f93c0038d00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93d40a1f90 0x7f93d4142370 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 --2- 172.21.15.12:0/4009595539 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f93d40a2360 0x7f93d41329b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 >> 172.21.15.12:0/4009595539 conn(0x7f93d40a3b50 msgr2=0x7f93d40a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 shutdown_connections 2023-09-29T23:13:42.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:42.766+0000 7f93da60e700 1 -- 172.21.15.12:0/4009595539 wait complete. 2023-09-29T23:13:45.779 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:45.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 10 monclient: get_monmap_and_config 2023-09-29T23:13:45.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 10 monclient: build_initial_monmap 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 10 monclient: monmap: 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:45.906064+0000 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:45.906064+0000 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:45.906 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 1 Processor -- start 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 1 -- start start 2023-09-29T23:13:45.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.905+0000 7f4be0456700 10 monclient: init 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient: _add_conns ranks=[6,3,7,5,9,1,8,10,4,2,0] 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-d con 0x7f4bd80a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80a2360 0x7f4bd80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-g con 0x7f4bd80a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd8133740 0x7f4bd8133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-i con 0x7f4bd8133740 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809e350 con 0x7f4bd80a2360 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809ef70 con 0x7f4bd80a1f90 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809da70 con 0x7f4bd8133740 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4be0456700 10 monclient(hunting): authenticate will time out at 1135.266555s 2023-09-29T23:13:45.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:47172/0 (socket says 172.21.15.12:47172) 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 1 -- 172.21.15.12:0/1369666594 learned_addr learned my addr 172.21.15.12:0/1369666594 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): get_auth_request con 0x7f4bd80a1f90 auth_method 0 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde9f3700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd8133740 0x7f4bd8133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bdd9f1700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80a2360 0x7f4bd80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde9f3700 10 monclient(hunting): get_auth_request con 0x7f4bd8133740 auth_method 0 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde9f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde9f3700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bde9f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bdd9f1700 10 monclient(hunting): get_auth_request con 0x7f4bd80a2360 auth_method 0 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bdd9f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bdd9f1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.906+0000 7f4bdd9f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 -- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd8133740 msgr2=0x7f4bd8133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd8133740 0x7f4bd8133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 -- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80a2360 msgr2=0x7f4bd80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80a2360 0x7f4bd80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 10 monclient: _finish_hunting 0 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 monclient: found mon.noname-d 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:45.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 -- 172.21.15.12:0/1369666594 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8068ed0 con 0x7f4bd80a1f90 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009a10 tx=0x7f4bc8009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=61c867b250ffbbff server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 1 -- 172.21.15.12:0/1369666594 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc800baf0 con 0x7f4bd80a1f90 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: dump: 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: _finish_auth 0 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 1 -- 172.21.15.12:0/1369666594 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc800bc70 con 0x7f4bd80a1f90 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 5 monclient: authenticate success, global_id 4389 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 4 set_mon_vals no callback set 2023-09-29T23:13:45.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 1 -- 172.21.15.12:0/1369666594 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc800bde0 con 0x7f4bd80a1f90 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4bdd1f0700 10 monclient: dump: 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:45.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 10 monclient: shutdown 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:45.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 1 -- 172.21.15.12:0/1369666594 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 msgr2=0x7f4bd809f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.907+0000 7f4be0456700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009a10 tx=0x7f4bc8009250 comp rx=0 tx=0).stop 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 -- 172.21.15.12:0/1369666594 shutdown_connections 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd8133740 0x7f4bd8133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809f5e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 --2- 172.21.15.12:0/1369666594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80a2360 0x7f4bd80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 -- 172.21.15.12:0/1369666594 >> 172.21.15.12:0/1369666594 conn(0x7f4bd80a3b50 msgr2=0x7f4bd80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 -- 172.21.15.12:0/1369666594 shutdown_connections 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 -- 172.21.15.12:0/1369666594 wait complete. 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 10 monclient: build_initial_monmap 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 10 monclient: monmap: 2023-09-29T23:13:45.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:45.928 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.908+0000 7f4be0456700 1 Processor -- start 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 -- start start 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient: init 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient: _add_conns ranks=[7,2,6,8,4,5,0,3,1] 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-i con 0x7f4bd80a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-b con 0x7f4bd80a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): picked mon.noname-d con 0x7f4bd8133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809ffc0 con 0x7f4bd80a2360 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd8130cf0 con 0x7f4bd8133740 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd812fb20 con 0x7f4bd80a1f90 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4be0456700 10 monclient(hunting): authenticate will time out at 1135.269508s 2023-09-29T23:13:45.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:47188/0 (socket says 172.21.15.12:47188) 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48624/0 (socket says 172.21.15.12:48624) 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 -- 172.21.15.12:0/3324161076 learned_addr learned my addr 172.21.15.12:0/3324161076 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): get_auth_request con 0x7f4bd8133740 auth_method 0 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): get_auth_request con 0x7f4bd80a2360 auth_method 0 2023-09-29T23:13:45.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): get_auth_request con 0x7f4bd80a1f90 auth_method 0 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde1f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 -- 172.21.15.12:0/3324161076 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd80a1f90 msgr2=0x7f4bd809c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 -- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 msgr2=0x7f4bd809ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient: _finish_hunting 0 2023-09-29T23:13:45.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 monclient: found mon.noname-d 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bde9f3700 1 -- 172.21.15.12:0/3324161076 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80094f0 con 0x7f4bd8133740 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.909+0000 7f4bdd9f1700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bde9f3700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009d70 tx=0x7f4bd000b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4bd0009ad0 con 0x7f4bd8133740 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: dump: 2023-09-29T23:13:45.934 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:45.935 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: _finish_auth 0 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000f040 con 0x7f4bd8133740 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 5 monclient: authenticate success, global_id 4398 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 10 monclient: _renew_subs 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:45.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd804c960 con 0x7f4bd8133740 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4bd0014900 con 0x7f4bd8133740 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bce7fc700 10 monclient: dump: 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:45.937 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:45.938 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4bdc9ef700 4 set_mon_vals no callback set 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 10 monclient: _renew_subs 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.910+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd809d210 con 0x7f4bd8133740 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4bd0014480 con 0x7f4bd8133740 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4be0456700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4be0456700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4be0456700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd8070bc0 con 0x7f4bd8133740 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4bce7fc700 1 --2- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 0x7f4bbc038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4bd0047a50 con 0x7f4bd8133740 2023-09-29T23:13:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 0x7f4bbc038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.911+0000 7f4bde1f2700 10 monclient: get_auth_request con 0x7f4bbc036780 auth_method 0 2023-09-29T23:13:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.912+0000 7f4bde1f2700 1 --2- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 0x7f4bbc038c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80099b0 tx=0x7f4bc8017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.916+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bd00112c0 con 0x7f4bd8133740 2023-09-29T23:13:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.916+0000 7f4bce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:45.916+0000 7f4bce7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:46.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4be0456700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:46.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4be0456700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4be0456700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4bd8066790 con 0x7f4bd8133740 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4bce7fc700 1 -- 172.21.15.12:0/3324161076 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4bd0018070 con 0x7f4bd8133740 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4bce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.150+0000 7f4bce7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:46.151 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 msgr2=0x7f4bbc038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 0x7f4bbc038c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80099b0 tx=0x7f4bc8017000 comp rx=0 tx=0).stop 2023-09-29T23:13:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 10 monclient: shutdown 2023-09-29T23:13:46.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 msgr2=0x7f4bd812d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009d70 tx=0x7f4bd000b040 comp rx=0 tx=0).stop 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 shutdown_connections 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4bd8133740 0x7f4bd812d2e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4bd80a1f90 0x7f4bd809c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> v2:172.21.15.153:6812/83257 conn(0x7f4bbc036780 0x7f4bbc038c30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 --2- 172.21.15.12:0/3324161076 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4bd80a2360 0x7f4bd809ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 >> 172.21.15.12:0/3324161076 conn(0x7f4bd80a3b50 msgr2=0x7f4bd8135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:46.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 shutdown_connections 2023-09-29T23:13:46.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:46.153+0000 7f4be0456700 1 -- 172.21.15.12:0/3324161076 wait complete. 2023-09-29T23:13:49.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.290+0000 7f8586297700 10 monclient: get_monmap_and_config 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.290+0000 7f8586297700 10 monclient: build_initial_monmap 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.290+0000 7f8586297700 10 monclient: monmap: 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:49.291728+0000 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:49.291728+0000 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:49.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:49.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.291+0000 7f8586297700 1 Processor -- start 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.291+0000 7f8586297700 1 -- start start 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.291+0000 7f8586297700 10 monclient: init 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient: _add_conns ranks=[9,4,10,7,5,3,2,1,6,0,8] 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): picked mon.noname-a con 0x7f8580133130 addr v2:172.21.15.12:3300/0 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): picked mon.noname-c con 0x7f8580133500 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f858012fc20 0x7f85801300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): picked mon.noname-f con 0x7f858012fc20 addr v2:172.21.15.153:3300/0 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85800a27a0 con 0x7f8580133130 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858009adf0 con 0x7f858012fc20 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f858009bd60 con 0x7f8580133500 2023-09-29T23:13:49.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8586297700 10 monclient(hunting): authenticate will time out at 1138.652240s 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42822/0 (socket says 172.21.15.12:42822) 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56066/0 (socket says 172.21.15.12:56066) 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 -- 172.21.15.12:0/2332714489 learned_addr learned my addr 172.21.15.12:0/2332714489 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): get_auth_request con 0x7f8580133500 auth_method 0 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): get_auth_request con 0x7f8580133130 auth_method 0 2023-09-29T23:13:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585a96700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.153:3300/0 conn(0x7f858012fc20 0x7f85801300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585a96700 10 monclient(hunting): get_auth_request con 0x7f858012fc20 auth_method 0 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585a96700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 -- 172.21.15.12:0/2332714489 >> v2:172.21.15.153:3300/0 conn(0x7f858012fc20 msgr2=0x7f85801300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.153:3300/0 conn(0x7f858012fc20 0x7f85801300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 -- 172.21.15.12:0/2332714489 >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 msgr2=0x7f85800a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient: _finish_hunting 0 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 monclient: found mon.noname-c 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 -- 172.21.15.12:0/2332714489 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580068ed0 con 0x7f8580133500 2023-09-29T23:13:49.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8585295700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f8584a94700 1 --2- 172.21.15.12:0/2332714489 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85800a7560 tx=0x7f8574009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=11aab300f4f19510 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.292+0000 7f857ffff700 1 -- 172.21.15.12:0/2332714489 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8574009e70 con 0x7f8580133500 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: dump: 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:49.307 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:49.309 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: _finish_auth 0 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 5 monclient: authenticate success, global_id 4369 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:49.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 1 -- 172.21.15.12:0/2332714489 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857400bce0 con 0x7f8580133500 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 4 set_mon_vals no callback set 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 1 -- 172.21.15.12:0/2332714489 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f857400be50 con 0x7f8580133500 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f857ffff700 10 monclient: dump: 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.311 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:49.312 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 10 monclient: shutdown 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 -- 172.21.15.12:0/2332714489 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 msgr2=0x7f8580133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 --2- 172.21.15.12:0/2332714489 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f85800a7560 tx=0x7f8574009250 comp rx=0 tx=0).stop 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 -- 172.21.15.12:0/2332714489 shutdown_connections 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.153:3300/0 conn(0x7f858012fc20 0x7f85801300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 --2- 172.21.15.12:0/2332714489 >> v2:172.21.15.12:3300/0 conn(0x7f8580133130 0x7f85800a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 --2- 172.21.15.12:0/2332714489 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133500 0x7f8580133950 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 -- 172.21.15.12:0/2332714489 >> 172.21.15.12:0/2332714489 conn(0x7f85800a6490 msgr2=0x7f85800a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 -- 172.21.15.12:0/2332714489 shutdown_connections 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.293+0000 7f8586297700 1 -- 172.21.15.12:0/2332714489 wait complete. 2023-09-29T23:13:49.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.294+0000 7f8586297700 10 monclient: build_initial_monmap 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.294+0000 7f8586297700 10 monclient: monmap: 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:49.314 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.294+0000 7f8586297700 1 Processor -- start 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.294+0000 7f8586297700 1 -- start start 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.294+0000 7f8586297700 10 monclient: init 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient: _add_conns ranks=[7,4,6,1,5,3,2,0,8] 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f858012fc20 0x7f858006d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): picked mon.noname-i con 0x7f858012fc20 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): picked mon.noname-c con 0x7f8580133130 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 0x7f858006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): picked mon.noname-d con 0x7f8580133500 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f858006f780 con 0x7f8580133130 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f858011def0 con 0x7f8580133500 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f858009b3e0 con 0x7f858012fc20 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:49.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8586297700 10 monclient(hunting): authenticate will time out at 1138.655197s 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42824/0 (socket says 172.21.15.12:42824) 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 -- 172.21.15.12:0/3207941304 learned_addr learned my addr 172.21.15.12:0/3207941304 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): get_auth_request con 0x7f8580133130 auth_method 0 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 0x7f858006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): get_auth_request con 0x7f8580133500 auth_method 0 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f858012fc20 0x7f858006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): get_auth_request con 0x7f858012fc20 auth_method 0 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:49.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 -- 172.21.15.12:0/3207941304 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f858012fc20 msgr2=0x7f858006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f858012fc20 0x7f858006d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 -- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 msgr2=0x7f858006dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 0x7f858006dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient: _finish_hunting 0 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 monclient: found mon.noname-c 2023-09-29T23:13:49.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 -- 172.21.15.12:0/3207941304 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578009420 con 0x7f8580133130 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8585a96700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 0x7f858006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f8584a94700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f858009bea0 tx=0x7f8574017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8574009e70 con 0x7f8580133130 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.295+0000 7f857d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: dump: 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.320 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:49.321 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: _finish_auth 0 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 5 monclient: authenticate success, global_id 4378 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857400be50 con 0x7f8580133130 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8574011d20 con 0x7f8580133130 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 10 monclient: _renew_subs 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85740094f0 con 0x7f8580133130 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:49.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857f7fe700 4 set_mon_vals no callback set 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f857d7fa700 10 monclient: dump: 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:49.323 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 10 monclient: _renew_subs 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.296+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858004c960 con 0x7f8580133130 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8574011420 con 0x7f8580133130 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8586297700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:49.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8586297700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8586297700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8580070bc0 con 0x7f8580133130 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f857d7fa700 1 --2- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 0x7f856c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f85740487b0 con 0x7f8580133130 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8585295700 1 --2- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 0x7f856c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8585295700 10 monclient: get_auth_request con 0x7f856c036880 auth_method 0 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.297+0000 7f8585295700 1 --2- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 0x7f856c038d30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8578009f40 tx=0x7f85780091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.302+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f857402d9a0 con 0x7f8580133130 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.302+0000 7f857d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:49.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.302+0000 7f857d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:49.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f8586297700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:49.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f8586297700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:49.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f8586297700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f858012fa00 con 0x7f8580133130 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f857d7fa700 1 -- 172.21.15.12:0/3207941304 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8574015070 con 0x7f8580133130 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f857d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.540+0000 7f857d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:49.541 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 msgr2=0x7f856c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 0x7f856c038d30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8578009f40 tx=0x7f85780091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 10 monclient: shutdown 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 msgr2=0x7f858006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f858009bea0 tx=0x7f8574017000 comp rx=0 tx=0).stop 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 shutdown_connections 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> v2:172.21.15.153:6812/83257 conn(0x7f856c036880 0x7f856c038d30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8580133500 0x7f858006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f858012fc20 0x7f858006d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 --2- 172.21.15.12:0/3207941304 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8580133130 0x7f858006d850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:49.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 >> 172.21.15.12:0/3207941304 conn(0x7f85800a6490 msgr2=0x7f8580130490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:49.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 shutdown_connections 2023-09-29T23:13:49.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:49.542+0000 7f8586297700 1 -- 172.21.15.12:0/3207941304 wait complete. 2023-09-29T23:13:52.555 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:52.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.685+0000 7f071508e700 10 monclient: get_monmap_and_config 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.685+0000 7f071508e700 10 monclient: build_initial_monmap 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.685+0000 7f071508e700 10 monclient: monmap: 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:52.686592+0000 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:52.686592+0000 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:52.686 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:52.687 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:52.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.686+0000 7f071508e700 1 Processor -- start 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.686+0000 7f071508e700 1 -- start start 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.686+0000 7f071508e700 10 monclient: init 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient: _add_conns ranks=[1,6,3,7,0,5,2,9,8,10,4] 2023-09-29T23:13:52.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0710133130 addr v1:172.21.15.153:6789/0 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): picked mon.noname-d con 0x7f0710133c10 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710138670 0x7f071013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): picked mon.noname-g con 0x7f0710138670 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0710068ed0 con 0x7f0710133130 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071009b2a0 con 0x7f0710138670 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f07100a27a0 con 0x7f0710133c10 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:52.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071508e700 10 monclient(hunting): authenticate will time out at 1142.047153s 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:47202/0 (socket says 172.21.15.12:47202) 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 -- 172.21.15.12:0/639124862 learned_addr learned my addr 172.21.15.12:0/639124862 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): get_auth_request con 0x7f0710133c10 auth_method 0 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:52.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071488d700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710138670 0x7f071013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071488d700 10 monclient(hunting): get_auth_request con 0x7f0710138670 auth_method 0 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071488d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071488d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f071488d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient(hunting): handle_auth_done global_id 4416 payload 274 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 -- 172.21.15.12:0/639124862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710138670 msgr2=0x7f071013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710138670 0x7f071013aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 -- 172.21.15.12:0/639124862 >> v1:172.21.15.153:6789/0 conn(0x7f0710133130 legacy=0x7f0710133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 monclient: found mon.noname-d 2023-09-29T23:13:52.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 -- 172.21.15.12:0/639124862 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071004c960 con 0x7f0710133c10 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070f7fe700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06f80099e0 tx=0x7f06f8009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1a7a75d251b12329 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070effd700 1 -- 172.21.15.12:0/639124862 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f06f800bce0 con 0x7f0710133c10 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.687+0000 7f070effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: dump: 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.700 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:52.701 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: _finish_auth 0 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 1 -- 172.21.15.12:0/639124862 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f800be60 con 0x7f0710133c10 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 5 monclient: authenticate success, global_id 4416 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:52.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 1 -- 172.21.15.12:0/639124862 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f06f80118a0 con 0x7f0710133c10 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 4 set_mon_vals no callback set 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f070effd700 10 monclient: dump: 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:52.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 10 monclient: shutdown 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:52.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 -- 172.21.15.12:0/639124862 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 msgr2=0x7f07100a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06f80099e0 tx=0x7f06f8009250 comp rx=0 tx=0).stop 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 -- 172.21.15.12:0/639124862 shutdown_connections 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710138670 0x7f071013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 --2- 172.21.15.12:0/639124862 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0710133c10 0x7f07100a0e80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 -- 172.21.15.12:0/639124862 >> 172.21.15.12:0/639124862 conn(0x7f07100a6490 msgr2=0x7f07100a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 -- 172.21.15.12:0/639124862 shutdown_connections 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.688+0000 7f071508e700 1 -- 172.21.15.12:0/639124862 wait complete. 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.689+0000 7f071508e700 10 monclient: build_initial_monmap 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.689+0000 7f071508e700 10 monclient: monmap: 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:52.705 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:52.706 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.689+0000 7f071508e700 1 Processor -- start 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.689+0000 7f071508e700 1 -- start start 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.689+0000 7f071508e700 10 monclient: init 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient: _add_conns ranks=[3,7,2,1,4,8,6,5,0] 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 0x7f0710137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): picked mon.noname-g con 0x7f0710133130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 0x7f071009dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): picked mon.noname-i con 0x7f0710133c10 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): picked mon.noname-b con 0x7f0710138670 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0710134ef0 con 0x7f0710138670 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071009b3e0 con 0x7f0710133130 2023-09-29T23:13:52.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f071009b020 con 0x7f0710133c10 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071508e700 10 monclient(hunting): authenticate will time out at 1142.050171s 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48626/0 (socket says 172.21.15.12:48626) 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 -- 172.21.15.12:0/1494415923 learned_addr learned my addr 172.21.15.12:0/1494415923 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): get_auth_request con 0x7f0710138670 auth_method 0 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 0x7f071009dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:52.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 0x7f0710137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): get_auth_request con 0x7f0710133c10 auth_method 0 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): get_auth_request con 0x7f0710133130 auth_method 0 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:52.710 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2023-09-29T23:13:52.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 -- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 msgr2=0x7f071009dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 0x7f071009dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 -- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 msgr2=0x7f0710137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 0x7f0710137ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient: _finish_hunting 0 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 monclient: found mon.noname-b 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 -- 172.21.15.12:0/1494415923 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80094f0 con 0x7f0710138670 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070f7fe700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 0x7f071009dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f070ffff700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 0x7f0710137ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:52.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f071488d700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0708007f20 tx=0x7f07080098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.690+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f070800eb90 con 0x7f0710138670 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: dump: 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:52.713 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: _finish_auth 0 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:52.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070800ed10 con 0x7f0710138670 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 5 monclient: authenticate success, global_id 4340 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 10 monclient: _renew_subs 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0710134be0 con 0x7f0710138670 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f070800ee80 con 0x7f0710138670 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f070e7fc700 4 set_mon_vals no callback set 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 10 monclient: dump: 2023-09-29T23:13:52.715 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:52.716 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 10 monclient: _renew_subs 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0710135840 con 0x7f0710138670 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.691+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f070801e460 con 0x7f0710138670 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f0707fff700 1 --2- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 0x7f06f0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0708047860 con 0x7f0710138670 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f071508e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:52.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f071508e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f071508e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0710070bc0 con 0x7f0710138670 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f070ffff700 1 --2- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 0x7f06f0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f070ffff700 10 monclient: get_auth_request con 0x7f06f0036850 auth_method 0 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.692+0000 7f070ffff700 1 --2- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 0x7f06f0038d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0700001d20 tx=0x7f0700009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.697+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f070801b900 con 0x7f0710138670 2023-09-29T23:13:52.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.697+0000 7f0707fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:52.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.697+0000 7f0707fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:52.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.931+0000 7f071508e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:52.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f071508e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:52.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f071508e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:13:52.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0710066790 con 0x7f0710138670 2023-09-29T23:13:52.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f0707fff700 1 -- 172.21.15.12:0/1494415923 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f070802c100 con 0x7f0710138670 2023-09-29T23:13:52.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f0707fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:52.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.932+0000 7f0707fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:52.933 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:52.933 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 msgr2=0x7f06f0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 0x7f06f0038d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0700001d20 tx=0x7f0700009040 comp rx=0 tx=0).stop 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 10 monclient: shutdown 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 msgr2=0x7f071009e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0708007f20 tx=0x7f07080098a0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 shutdown_connections 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> v2:172.21.15.153:6812/83257 conn(0x7f06f0036850 0x7f06f0038d00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0710138670 0x7f071009e200 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0710133130 0x7f0710137ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 --2- 172.21.15.12:0/1494415923 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0710133c10 0x7f071009dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:52.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 >> 172.21.15.12:0/1494415923 conn(0x7f07100a6490 msgr2=0x7f071012e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:52.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 shutdown_connections 2023-09-29T23:13:52.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:52.934+0000 7f071508e700 1 -- 172.21.15.12:0/1494415923 wait complete. 2023-09-29T23:13:55.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 10 monclient: get_monmap_and_config 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 10 monclient: build_initial_monmap 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 10 monclient: monmap: 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:56.079 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:56.079966+0000 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:56.079966+0000 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:56.080 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:56.081 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:56.081 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 1 Processor -- start 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 1 -- start start 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.079+0000 7fcc397ba700 10 monclient: init 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient: _add_conns ranks=[5,4,3,10,9,2,7,0,8,6,1] 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fcc3406f2a0 0x7fcc3406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-h con 0x7fcc3406f2a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-c con 0x7fcc340969e0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc34139df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-g con 0x7fcc340972f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3406bd10 con 0x7fcc340972f0 2023-09-29T23:13:56.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3406ad20 con 0x7fcc340969e0 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3409fd90 con 0x7fcc3406f2a0 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc397ba700 10 monclient(hunting): authenticate will time out at 1145.440312s 2023-09-29T23:13:56.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42384/0 (socket says 172.21.15.12:42384) 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 -- 172.21.15.12:0/2826552646 learned_addr learned my addr 172.21.15.12:0/2826552646 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): get_auth_request con 0x7fcc340969e0 auth_method 0 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fcc3406f2a0 0x7fcc3406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): get_auth_request con 0x7fcc3406f2a0 auth_method 0 2023-09-29T23:13:56.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc34139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): get_auth_request con 0x7fcc340972f0 auth_method 0 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 -- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fcc3406f2a0 msgr2=0x7fcc3406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:56.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fcc3406f2a0 0x7fcc3406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 -- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 msgr2=0x7fcc34139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc34139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 monclient: found mon.noname-c 2023-09-29T23:13:56.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:13:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 -- 172.21.15.12:0/2826552646 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34068ed0 con 0x7fcc340969e0 2023-09-29T23:13:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.080+0000 7fcc327fc700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc280099e0 tx=0x7fcc28009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9e2a5893639382eb server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 1 -- 172.21.15.12:0/2826552646 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc2800bb00 con 0x7fcc340969e0 2023-09-29T23:13:56.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: dump: 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:56.102 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:56.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 5 monclient: authenticate success, global_id 4387 2023-09-29T23:13:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 1 -- 172.21.15.12:0/2826552646 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800bc80 con 0x7fcc340969e0 2023-09-29T23:13:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:56.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 4 set_mon_vals no callback set 2023-09-29T23:13:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 1 -- 172.21.15.12:0/2826552646 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc2800bdf0 con 0x7fcc340969e0 2023-09-29T23:13:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:13:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc31ffb700 10 monclient: dump: 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:56.106 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:56.107 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 10 monclient: shutdown 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 -- 172.21.15.12:0/2826552646 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 msgr2=0x7fcc34096db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc280099e0 tx=0x7fcc28009250 comp rx=0 tx=0).stop 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 -- 172.21.15.12:0/2826552646 shutdown_connections 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc34139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fcc3406f2a0 0x7fcc3406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 --2- 172.21.15.12:0/2826552646 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc340969e0 0x7fcc34096db0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 -- 172.21.15.12:0/2826552646 >> 172.21.15.12:0/2826552646 conn(0x7fcc34063e80 msgr2=0x7fcc340a0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 -- 172.21.15.12:0/2826552646 shutdown_connections 2023-09-29T23:13:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.081+0000 7fcc397ba700 1 -- 172.21.15.12:0/2826552646 wait complete. 2023-09-29T23:13:56.110 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.082+0000 7fcc397ba700 10 monclient: build_initial_monmap 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.082+0000 7fcc397ba700 10 monclient: monmap: 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:56.111 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:56.112 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:56.113 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:56.113 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:56.113 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:56.113 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:56.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.082+0000 7fcc397ba700 1 Processor -- start 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.082+0000 7fcc397ba700 1 -- start start 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.082+0000 7fcc397ba700 10 monclient: init 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient: _add_conns ranks=[2,8,3,0,5,4,6,7,1] 2023-09-29T23:13:56.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-b con 0x7fcc3406f2a0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-e con 0x7fcc340969e0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc3409c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): picked mon.noname-g con 0x7fcc340972f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:56.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3406f910 con 0x7fcc3406f2a0 2023-09-29T23:13:56.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34120370 con 0x7fcc340972f0 2023-09-29T23:13:56.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:56.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fcc341432c0 con 0x7fcc340969e0 2023-09-29T23:13:56.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:56.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc397ba700 10 monclient(hunting): authenticate will time out at 1145.443313s 2023-09-29T23:13:56.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56304/0 (socket says 172.21.15.12:56304) 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:42504/0 (socket says 172.21.15.12:42504) 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 1 -- 172.21.15.12:0/1027373695 learned_addr learned my addr 172.21.15.12:0/1027373695 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): get_auth_request con 0x7fcc3406f2a0 auth_method 0 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): get_auth_request con 0x7fcc340969e0 auth_method 0 2023-09-29T23:13:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc3409c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): get_auth_request con 0x7fcc340972f0 auth_method 0 2023-09-29T23:13:56.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:56.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:56.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:56.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 -- 172.21.15.12:0/1027373695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 msgr2=0x7fcc3409c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:56.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc3409c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 -- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 msgr2=0x7fcc34142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient: _finish_hunting 0 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 monclient: found mon.noname-e 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 -- 172.21.15.12:0/1027373695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280094f0 con 0x7fcc340969e0 2023-09-29T23:13:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc32ffd700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.083+0000 7fcc327fc700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcc280099b0 tx=0x7fcc28017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc28006b30 con 0x7fcc340969e0 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: dump: 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:56.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:56.125 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:56.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: _finish_auth 0 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2801a430 con 0x7fcc340969e0 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 5 monclient: authenticate success, global_id 4328 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 10 monclient: _renew_subs 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3404c960 con 0x7fcc340969e0 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:56.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc2801aa20 con 0x7fcc340969e0 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc317fa700 4 set_mon_vals no callback set 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc1affd700 10 monclient: dump: 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 10 monclient: _renew_subs 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.084+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3409e130 con 0x7fcc340969e0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fcc2801a5a0 con 0x7fcc340969e0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc397ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc397ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc397ba700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc34070bc0 con 0x7fcc340969e0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc1affd700 1 --2- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 0x7fcc20038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcc28048b60 con 0x7fcc340969e0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc32ffd700 1 --2- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 0x7fcc20038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.085+0000 7fcc32ffd700 10 monclient: get_auth_request con 0x7fcc20036850 auth_method 0 2023-09-29T23:13:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.086+0000 7fcc32ffd700 1 --2- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 0x7fcc20038d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009fd0 tx=0x7fcc240091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:56.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.090+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc280193b0 con 0x7fcc340969e0 2023-09-29T23:13:56.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.090+0000 7fcc1affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:56.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.090+0000 7fcc1affd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc397ba700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc397ba700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:56.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc397ba700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc3409b0a0 con 0x7fcc340969e0 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc1affd700 1 -- 172.21.15.12:0/1027373695 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc28015070 con 0x7fcc340969e0 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc1affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.335+0000 7fcc1affd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:56.336 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 msgr2=0x7fcc20038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:56.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 0x7fcc20038d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009fd0 tx=0x7fcc240091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 10 monclient: shutdown 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 msgr2=0x7fcc34132f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcc280099b0 tx=0x7fcc28017000 comp rx=0 tx=0).stop 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 shutdown_connections 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> v2:172.21.15.153:6812/83257 conn(0x7fcc20036850 0x7fcc20038d00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc340972f0 0x7fcc3409c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc3406f2a0 0x7fcc34142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 --2- 172.21.15.12:0/1027373695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc340969e0 0x7fcc34132f70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 >> 172.21.15.12:0/1027373695 conn(0x7fcc34063e80 msgr2=0x7fcc340a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 shutdown_connections 2023-09-29T23:13:56.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:56.337+0000 7fcc397ba700 1 -- 172.21.15.12:0/1027373695 wait complete. 2023-09-29T23:13:59.349 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:13:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 10 monclient: get_monmap_and_config 2023-09-29T23:13:59.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 10 monclient: build_initial_monmap 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 10 monclient: monmap: 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:59.479016+0000 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:59.479016+0000 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:13:59.479 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 1 Processor -- start 2023-09-29T23:13:59.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 1 -- start start 2023-09-29T23:13:59.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.478+0000 7f3021861700 10 monclient: init 2023-09-29T23:13:59.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:59.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient: _add_conns ranks=[6,8,2,1,3,0,9,7,10,5,4] 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): picked mon.noname-d con 0x7f301c130fe0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f301c06f0b0 0x7f301c06f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): picked mon.noname-e con 0x7f301c06f0b0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): picked mon.noname-b con 0x7f301c0969e0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c1313b0 con 0x7f301c0969e0 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f301c131e00 con 0x7f301c130fe0 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f301c06d690 con 0x7f301c06f0b0 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f3021861700 10 monclient(hunting): authenticate will time out at 1148.839293s 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:40358/0 (socket says 172.21.15.12:40358) 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56332/0 (socket says 172.21.15.12:56332) 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 -- 172.21.15.12:0/351405973 learned_addr learned my addr 172.21.15.12:0/351405973 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:59.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): get_auth_request con 0x7f301c130fe0 auth_method 0 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): get_auth_request con 0x7f301c0969e0 auth_method 0 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f301c06f0b0 0x7f301c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): get_auth_request con 0x7f301c06f0b0 auth_method 0 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2023-09-29T23:13:59.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 -- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f301c06f0b0 msgr2=0x7f301c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f301c06f0b0 0x7f301c06f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 -- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 msgr2=0x7f301c096e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient: _finish_hunting 0 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 monclient: found mon.noname-d 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301affd700 1 -- 172.21.15.12:0/351405973 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c068ed0 con 0x7f301c130fe0 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.479+0000 7f301b7fe700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f301affd700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f300c009f50 tx=0x7f300c009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f20b89b45ed0f617 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:59.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 1 -- 172.21.15.12:0/351405973 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f300c00e070 con 0x7f301c130fe0 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: dump: 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:59.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: _finish_auth 0 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:59.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 1 -- 172.21.15.12:0/351405973 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c00bdd0 con 0x7f301c130fe0 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 5 monclient: authenticate success, global_id 4425 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 10 monclient: get_monmap_and_config success 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 1 -- 172.21.15.12:0/351405973 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f300c0092e0 con 0x7f301c130fe0 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 4 set_mon_vals no callback set 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3019ffb700 10 monclient: dump: 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:59.496 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:59.497 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 10 monclient: shutdown 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 -- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 msgr2=0x7f301c131740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f300c009f50 tx=0x7f300c009840 comp rx=0 tx=0).stop 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 -- 172.21.15.12:0/351405973 shutdown_connections 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f301c0969e0 0x7f301c096e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c131740 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 --2- 172.21.15.12:0/351405973 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f301c06f0b0 0x7f301c06f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 -- 172.21.15.12:0/351405973 >> 172.21.15.12:0/351405973 conn(0x7f301c0a4430 msgr2=0x7f301c0a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 -- 172.21.15.12:0/351405973 shutdown_connections 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.480+0000 7f3021861700 1 -- 172.21.15.12:0/351405973 wait complete. 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.481+0000 7f3021861700 10 monclient: build_initial_monmap 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.481+0000 7f3021861700 10 monclient: monmap: 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:13:59.499 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.481+0000 7f3021861700 1 Processor -- start 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.481+0000 7f3021861700 1 -- start start 2023-09-29T23:13:59.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.481+0000 7f3021861700 10 monclient: init 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient: _reopen_session rank -1 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient: _add_conns ranks=[4,3,6,0,5,8,7,2,1] 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): picked mon.noname-c con 0x7f301c06f0b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 0x7f301c11c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): picked mon.noname-g con 0x7f301c0969e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c11c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): picked mon.noname-d con 0x7f301c130fe0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c11fd40 con 0x7f301c0969e0 2023-09-29T23:13:59.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0a0550 con 0x7f301c06f0b0 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): start opening mon connection 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0a0180 con 0x7f301c130fe0 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): _renew_subs 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f3021861700 10 monclient(hunting): authenticate will time out at 1148.842330s 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42408/0 (socket says 172.21.15.12:42408) 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 1 -- 172.21.15.12:0/1442888984 learned_addr learned my addr 172.21.15.12:0/1442888984 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): get_auth_request con 0x7f301c06f0b0 auth_method 0 2023-09-29T23:13:59.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c11c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): get_auth_request con 0x7f301c130fe0 auth_method 0 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:13:59.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 0x7f301c11c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 10 monclient(hunting): get_auth_request con 0x7f301c0969e0 auth_method 0 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 -- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 msgr2=0x7f301c12f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 -- 172.21.15.12:0/1442888984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 msgr2=0x7f301c11c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 0x7f301c11c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 monclient: found mon.noname-d 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301b7fe700 1 -- 172.21.15.12:0/1442888984 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3014009420 con 0x7f301c130fe0 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301a7fc700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 0x7f301c11c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.482+0000 7f301affd700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f301b7fe700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c11c560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f301400ea80 tx=0x7f30140091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3014009670 con 0x7f301c130fe0 2023-09-29T23:13:59.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: dump: 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:59.506 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: _finish_auth 0 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301400ccd0 con 0x7f301c130fe0 2023-09-29T23:13:59.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 5 monclient: authenticate success, global_id 4434 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 10 monclient: _renew_subs 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c009ae0 con 0x7f301c130fe0 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f301400ce40 con 0x7f301c130fe0 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f30197fa700 4 set_mon_vals no callback set 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3012ffd700 10 monclient: dump: 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:13:59.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:13:59.509 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 10 monclient: _renew_subs 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.483+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c04c960 con 0x7f301c130fe0 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f301400c850 con 0x7f301c130fe0 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3021861700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:59.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3021861700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3021861700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301c11fba0 con 0x7f301c130fe0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3012ffd700 1 --2- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 0x7f2ffc038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3014047260 con 0x7f301c130fe0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f301affd700 1 --2- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 0x7f2ffc038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.484+0000 7f301affd700 10 monclient: get_auth_request con 0x7f2ffc036850 auth_method 0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.485+0000 7f301affd700 1 --2- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 0x7f2ffc038d00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f300c009df0 tx=0x7f300c018000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.488+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f301c11fba0 con 0x7f301c130fe0 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.489+0000 7f3012ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:13:59.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.489+0000 7f3012ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:13:59.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3021861700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:13:59.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3021861700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:59.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3021861700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:13:59.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f301c11fba0 con 0x7f301c130fe0 2023-09-29T23:13:59.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3012ffd700 1 -- 172.21.15.12:0/1442888984 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f301c11fba0 con 0x7f301c130fe0 2023-09-29T23:13:59.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3012ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:13:59.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.723+0000 7f3012ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:13:59.724 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:13:59.724 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:13:59.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 msgr2=0x7f2ffc038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 0x7f2ffc038d00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f300c009df0 tx=0x7f300c018000 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 10 monclient: shutdown 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 msgr2=0x7f301c11c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c11c560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f301400ea80 tx=0x7f30140091a0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 shutdown_connections 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f301c130fe0 0x7f301c11c560 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f301c06f0b0 0x7f301c12f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> v2:172.21.15.153:6812/83257 conn(0x7f2ffc036850 0x7f2ffc038d00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 --2- 172.21.15.12:0/1442888984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f301c0969e0 0x7f301c11c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:13:59.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 >> 172.21.15.12:0/1442888984 conn(0x7f301c0a4430 msgr2=0x7f301c09d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:13:59.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 shutdown_connections 2023-09-29T23:13:59.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:13:59.725+0000 7f3021861700 1 -- 172.21.15.12:0/1442888984 wait complete. 2023-09-29T23:14:02.738 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.866+0000 7fdccd646700 10 monclient: get_monmap_and_config 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.866+0000 7fdccd646700 10 monclient: build_initial_monmap 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.866+0000 7fdccd646700 10 monclient: monmap: 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:02.868597+0000 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:02.868597+0000 2023-09-29T23:14:02.868 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:02.869 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.867+0000 7fdccd646700 1 Processor -- start 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.867+0000 7fdccd646700 1 -- start start 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.867+0000 7fdccd646700 10 monclient: init 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient: _add_conns ranks=[7,2,9,5,8,10,1,4,0,3,6] 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdcc8133130 0x7fdcc80a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-i con 0x7fdcc8133130 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:02.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 0x7fdcc8133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-b con 0x7fdcc8133500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-a con 0x7fdcc812fc20 addr v2:172.21.15.12:3300/0 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc80a27a0 con 0x7fdcc812fc20 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc809adf0 con 0x7fdcc8133500 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc809bd60 con 0x7fdcc8133130 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:02.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccd646700 10 monclient(hunting): authenticate will time out at 1152.229303s 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 0x7fdcc8133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60020/0 (socket says 172.21.15.12:60020) 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 1 -- 172.21.15.12:0/47319743 learned_addr learned my addr 172.21.15.12:0/47319743 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): get_auth_request con 0x7fdcc8133500 auth_method 0 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): get_auth_request con 0x7fdcc812fc20 auth_method 0 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdcc8133130 0x7fdcc80a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): get_auth_request con 0x7fdcc8133130 auth_method 0 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 -- 172.21.15.12:0/47319743 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdcc8133130 msgr2=0x7fdcc80a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdcc8133130 0x7fdcc80a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 -- 172.21.15.12:0/47319743 >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 msgr2=0x7fdcc8130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/47319743 >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient: _finish_hunting 0 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 monclient: found mon.noname-b 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:02.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 -- 172.21.15.12:0/47319743 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8068ed0 con 0x7fdcc8133500 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdccce45700 1 --2- 172.21.15.12:0/47319743 >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.868+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 0x7fdcc8133950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009910 tx=0x7fdcb80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285f1caf572c844b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 1 -- 172.21.15.12:0/47319743 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb8011ab0 con 0x7fdcc8133500 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: dump: 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.885 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: _finish_auth 0 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 1 -- 172.21.15.12:0/47319743 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb8011c30 con 0x7fdcc8133500 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 5 monclient: authenticate success, global_id 4367 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 1 -- 172.21.15.12:0/47319743 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb8011da0 con 0x7fdcc8133500 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdcc6ffd700 10 monclient: dump: 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:02.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:02.888 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 4 set_mon_vals no callback set 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 10 monclient: shutdown 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 -- 172.21.15.12:0/47319743 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 msgr2=0x7fdcc8133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 0x7fdcc8133950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009910 tx=0x7fdcb80091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 -- 172.21.15.12:0/47319743 shutdown_connections 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 --2- 172.21.15.12:0/47319743 >> v2:172.21.15.12:3300/0 conn(0x7fdcc812fc20 0x7fdcc8130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdcc8133130 0x7fdcc80a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 --2- 172.21.15.12:0/47319743 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdcc8133500 0x7fdcc8133950 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 -- 172.21.15.12:0/47319743 >> 172.21.15.12:0/47319743 conn(0x7fdcc80a6490 msgr2=0x7fdcc80a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 -- 172.21.15.12:0/47319743 shutdown_connections 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.869+0000 7fdccd646700 1 -- 172.21.15.12:0/47319743 wait complete. 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.870+0000 7fdccd646700 10 monclient: build_initial_monmap 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.870+0000 7fdccd646700 10 monclient: monmap: 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:02.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.870+0000 7fdccd646700 1 Processor -- start 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.870+0000 7fdccd646700 1 -- start start 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.870+0000 7fdccd646700 10 monclient: init 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient: _add_conns ranks=[5,1,3,2,6,0,7,8,4] 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 0x7fdcc812e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-h con 0x7fdcc812fc20 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-f con 0x7fdcc8133130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc8133500 0x7fdcc812ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): picked mon.noname-g con 0x7fdcc8133500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc806eaa0 con 0x7fdcc8133130 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc806d560 con 0x7fdcc8133500 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc809b3e0 con 0x7fdcc812fc20 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccd646700 10 monclient(hunting): authenticate will time out at 1152.232342s 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 0x7fdcc812e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc8133500 0x7fdcc812ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:34224/0 (socket says 172.21.15.12:34224) 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 1 -- 172.21.15.12:0/1836642415 learned_addr learned my addr 172.21.15.12:0/1836642415 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): get_auth_request con 0x7fdcc8133130 auth_method 0 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): get_auth_request con 0x7fdcc812fc20 auth_method 0 2023-09-29T23:14:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): get_auth_request con 0x7fdcc8133500 auth_method 0 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.871+0000 7fdccce45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 -- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc8133500 msgr2=0x7fdcc812ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc8133500 0x7fdcc812ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 -- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 msgr2=0x7fdcc812e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 monclient: found mon.noname-h 2023-09-29T23:14:02.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 -- 172.21.15.12:0/1836642415 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0009ae0 con 0x7fdcc812fc20 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc7fff700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 0x7fdcc812e260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000ea80 tx=0x7fdcb00091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb00095d0 con 0x7fdcc812fc20 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: dump: 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.896 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: _finish_auth 0 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb000cc70 con 0x7fdcc812fc20 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 5 monclient: authenticate success, global_id 4316 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 10 monclient: _renew_subs 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb8009420 con 0x7fdcc812fc20 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb000cde0 con 0x7fdcc812fc20 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:14:02.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcbffff700 10 monclient: dump: 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:02.899 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdcc67fc700 4 set_mon_vals no callback set 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 10 monclient: _renew_subs 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.872+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc804c960 con 0x7fdcc812fc20 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fdcb000c7f0 con 0x7fdcc812fc20 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdccd646700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdccd646700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdccd646700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:14:02.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc8070bc0 con 0x7fdcc812fc20 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdcbffff700 1 --2- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 0x7fdca8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdcb00477c0 con 0x7fdcc812fc20 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 0x7fdca8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.873+0000 7fdcc77fe700 10 monclient: get_auth_request con 0x7fdca8036850 auth_method 0 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.874+0000 7fdcc77fe700 1 --2- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 0x7fdca8038d00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009910 tx=0x7fdcb8017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.878+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcb001b3b0 con 0x7fdcc812fc20 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.878+0000 7fdcbffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:02.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:02.878+0000 7fdcbffff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:03.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.115+0000 7fdccd646700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.115+0000 7fdccd646700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.115+0000 7fdccd646700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.115+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdcc806d030 con 0x7fdcc812fc20 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.116+0000 7fdcbffff700 1 -- 172.21.15.12:0/1836642415 <== mon.5 v2:172.21.15.153:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdcb001ab50 con 0x7fdcc812fc20 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.116+0000 7fdcbffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.116+0000 7fdcbffff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:03.117 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:03.118 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 msgr2=0x7fdca8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 0x7fdca8038d00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009910 tx=0x7fdcb8017000 comp rx=0 tx=0).stop 2023-09-29T23:14:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 10 monclient: shutdown 2023-09-29T23:14:03.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 msgr2=0x7fdcc812e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 0x7fdcc812e260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000ea80 tx=0x7fdcb00091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 shutdown_connections 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> v2:172.21.15.153:6812/83257 conn(0x7fdca8036850 0x7fdca8038d00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcc8133500 0x7fdcc812ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdcc812fc20 0x7fdcc812e260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 --2- 172.21.15.12:0/1836642415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcc8133130 0x7fdcc812e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 >> 172.21.15.12:0/1836642415 conn(0x7fdcc80a6490 msgr2=0x7fdcc8136130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 shutdown_connections 2023-09-29T23:14:03.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:03.118+0000 7fdccd646700 1 -- 172.21.15.12:0/1836642415 wait complete. 2023-09-29T23:14:06.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.260+0000 7f4b0b4fd700 10 monclient: get_monmap_and_config 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.260+0000 7f4b0b4fd700 10 monclient: build_initial_monmap 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.260+0000 7f4b0b4fd700 10 monclient: monmap: 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:06.262744+0000 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:06.262744+0000 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:06.262 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:06.263 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.261+0000 7f4b0b4fd700 1 Processor -- start 2023-09-29T23:14:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.261+0000 7f4b0b4fd700 1 -- start start 2023-09-29T23:14:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.261+0000 7f4b0b4fd700 10 monclient: init 2023-09-29T23:14:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:06.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient: _add_conns ranks=[7,2,4,0,5,10,3,1,8,6,9] 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4b04133130 0x7f4b040a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-i con 0x7f4b04133130 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 0x7f4b04133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-b con 0x7f4b04133500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-c con 0x7f4b0412fc20 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040a27a0 con 0x7f4b04133500 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0409adf0 con 0x7f4b0412fc20 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0409bd60 con 0x7f4b04133130 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0b4fd700 10 monclient(hunting): authenticate will time out at 1155.623330s 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 0x7f4b04133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:35064/0 (socket says 172.21.15.12:35064) 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 -- 172.21.15.12:0/3292134229 learned_addr learned my addr 172.21.15.12:0/3292134229 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): get_auth_request con 0x7f4b0412fc20 auth_method 0 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): get_auth_request con 0x7f4b04133500 auth_method 0 2023-09-29T23:14:06.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0a4fb700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4b04133130 0x7f4b040a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0a4fb700 10 monclient(hunting): get_auth_request con 0x7f4b04133130 auth_method 0 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0a4fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0a4fb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0a4fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 -- 172.21.15.12:0/3292134229 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4b04133130 msgr2=0x7f4b040a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4b04133130 0x7f4b040a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 -- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 msgr2=0x7f4b04133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 0x7f4b04133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient: _finish_hunting 0 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 monclient: found mon.noname-c 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 -- 172.21.15.12:0/3292134229 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04068ed0 con 0x7f4b0412fc20 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b09cfa700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 0x7f4b04133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:06.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.262+0000 7f4b0acfc700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009f80 tx=0x7f4afc009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c82683c790948bf6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 1 -- 172.21.15.12:0/3292134229 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4afc00e070 con 0x7f4b0412fc20 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: dump: 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: _finish_auth 0 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:06.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 5 monclient: authenticate success, global_id 4405 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 1 -- 172.21.15.12:0/3292134229 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc007400 con 0x7f4b0412fc20 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 4 set_mon_vals no callback set 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 1 -- 172.21.15.12:0/3292134229 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4afc0092e0 con 0x7f4b0412fc20 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b094f9700 10 monclient: dump: 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:06.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:06.280 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 10 monclient: shutdown 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 msgr2=0x7f4b04130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009f80 tx=0x7f4afc009840 comp rx=0 tx=0).stop 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/3292134229 shutdown_connections 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b04130090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4b04133130 0x7f4b040a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/3292134229 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4b04133500 0x7f4b04133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/3292134229 >> 172.21.15.12:0/3292134229 conn(0x7f4b040a6490 msgr2=0x7f4b040a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/3292134229 shutdown_connections 2023-09-29T23:14:06.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.263+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/3292134229 wait complete. 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.264+0000 7f4b0b4fd700 10 monclient: build_initial_monmap 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.264+0000 7f4b0b4fd700 10 monclient: monmap: 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.264+0000 7f4b0b4fd700 1 Processor -- start 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.264+0000 7f4b0b4fd700 1 -- start start 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.264+0000 7f4b0b4fd700 10 monclient: init 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient: _add_conns ranks=[4,6,1,2,3,8,5,7,0] 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-c con 0x7f4b0412fc20 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 0x7f4b04142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-d con 0x7f4b04133130 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04133500 0x7f4b0406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): picked mon.noname-f con 0x7f4b04133500 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0409b020 con 0x7f4b04133500 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0406ee80 con 0x7f4b0412fc20 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0411da00 con 0x7f4b04133130 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:06.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0b4fd700 10 monclient(hunting): authenticate will time out at 1155.626222s 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:35070/0 (socket says 172.21.15.12:35070) 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 -- 172.21.15.12:0/1601117505 learned_addr learned my addr 172.21.15.12:0/1601117505 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): get_auth_request con 0x7f4b0412fc20 auth_method 0 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 0x7f4b04142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): get_auth_request con 0x7f4b04133130 auth_method 0 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04133500 0x7f4b0406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): get_auth_request con 0x7f4b04133500 auth_method 0 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:06.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0acfc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 -- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 msgr2=0x7f4b04142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 0x7f4b04142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 -- 172.21.15.12:0/1601117505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04133500 msgr2=0x7f4b0406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04133500 0x7f4b0406d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient: _finish_hunting 0 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 monclient: found mon.noname-c 2023-09-29T23:14:06.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 -- 172.21.15.12:0/1601117505 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4009420 con 0x7f4b0412fc20 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b09cfa700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 0x7f4b04142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4b0a4fb700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000d890 tx=0x7f4b00008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4b00007470 con 0x7f4b0412fc20 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.265+0000 7f4afa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: dump: 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.288 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00012a70 con 0x7f4b0412fc20 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4b00012be0 con 0x7f4b0412fc20 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 5 monclient: authenticate success, global_id 4414 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _renew_subs 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b08cf8700 4 set_mon_vals no callback set 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc009ae0 con 0x7f4b0412fc20 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:06.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 10 monclient: dump: 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:06.291 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _renew_subs 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0404c960 con 0x7f4b0412fc20 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4b000125f0 con 0x7f4b0412fc20 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.266+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0411e0e0 con 0x7f4b0412fc20 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.267+0000 7f4afa7fc700 1 --2- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 0x7f4ae8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.267+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4b000477d0 con 0x7f4b0412fc20 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.267+0000 7f4b09cfa700 1 --2- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 0x7f4ae8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.267+0000 7f4b09cfa700 10 monclient: get_auth_request con 0x7f4ae8036850 auth_method 0 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.267+0000 7f4b09cfa700 1 --2- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 0x7f4ae8038d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b0406b580 tx=0x7f4af40091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.271+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b000128a0 con 0x7f4b0412fc20 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.272+0000 7f4afa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:06.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.272+0000 7f4afa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:06.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4b0b4fd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:06.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4b0b4fd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:06.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4b0b4fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:06.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4b04070bc0 con 0x7f4b0412fc20 2023-09-29T23:14:06.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4afa7fc700 1 -- 172.21.15.12:0/1601117505 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4b00011340 con 0x7f4b0412fc20 2023-09-29T23:14:06.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4afa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:06.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.510+0000 7f4afa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:06.512 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:06.512 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:06.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 msgr2=0x7f4ae8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:06.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 0x7f4ae8038d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b0406b580 tx=0x7f4af40091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 10 monclient: shutdown 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 msgr2=0x7f4b041426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000d890 tx=0x7f4b00008040 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 shutdown_connections 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> v2:172.21.15.153:6812/83257 conn(0x7f4ae8036850 0x7f4ae8038d00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04133500 0x7f4b0406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4b0412fc20 0x7f4b041426e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 --2- 172.21.15.12:0/1601117505 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f4b04133130 0x7f4b04142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 >> 172.21.15.12:0/1601117505 conn(0x7f4b040a6490 msgr2=0x7f4b040a4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:06.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 shutdown_connections 2023-09-29T23:14:06.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:06.512+0000 7f4b0b4fd700 1 -- 172.21.15.12:0/1601117505 wait complete. 2023-09-29T23:14:09.525 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.656+0000 7f08b8bd5700 10 monclient: get_monmap_and_config 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.656+0000 7f08b8bd5700 10 monclient: build_initial_monmap 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.657+0000 7f08b8bd5700 10 monclient: monmap: 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:09.658867+0000 2023-09-29T23:14:09.658 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:09.658867+0000 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:09.659 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:09.660 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:09.660 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.657+0000 7f08b8bd5700 1 Processor -- start 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.657+0000 7f08b8bd5700 1 -- start start 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.657+0000 7f08b8bd5700 10 monclient: init 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient: _add_conns ranks=[0,6,4,9,2,5,10,1,8,7,3] 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08b4099660 addr v1:172.21.15.12:6789/0 2023-09-29T23:14:09.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f08b409a870 0x7f08b4099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-d con 0x7f08b409a870 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 0x7f08b4137570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-c con 0x7f08b4099f70 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b4068ed0 con 0x7f08b4099660 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409bc50 con 0x7f08b4099f70 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40966a0 con 0x7f08b409a870 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b8bd5700 10 monclient(hunting): authenticate will time out at 1159.019368s 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b259c700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f08b4099660 0x7f08b4096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51816/0 (socket says 172.21.15.12:51816) 2023-09-29T23:14:09.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f08b409a870 0x7f08b4099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b259c700 1 -- 172.21.15.12:0/4029530701 learned_addr learned my addr 172.21.15.12:0/4029530701 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 0x7f08b4137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): get_auth_request con 0x7f08b409a870 auth_method 0 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): get_auth_request con 0x7f08b4099f70 auth_method 0 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:09.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 -- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f08b409a870 msgr2=0x7f08b4099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 --2- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f08b409a870 0x7f08b4099a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 -- 172.21.15.12:0/4029530701 >> v1:172.21.15.12:6789/0 conn(0x7f08b4099660 legacy=0x7f08b4096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 monclient: found mon.noname-c 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:09.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b159a700 1 -- 172.21.15.12:0/4029530701 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98820 0 0) 0x7f08b4068ed0 con 0x7f08b4099660 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b2d9d700 1 -- 172.21.15.12:0/4029530701 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b404c960 con 0x7f08b4099f70 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.658+0000 7f08b159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b2d9d700 1 --2- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 0x7f08b4137570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009e90 tx=0x7f08ac009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ce84f226ba5b16f4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 1 -- 172.21.15.12:0/4029530701 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f08ac00e070 con 0x7f08b4099f70 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: dump: 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.672 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:09.673 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: _finish_auth 0 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 5 monclient: authenticate success, global_id 4423 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 1 -- 172.21.15.12:0/4029530701 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac00bc30 con 0x7f08b4099f70 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 1 -- 172.21.15.12:0/4029530701 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f08ac00bda0 con 0x7f08b4099f70 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 4 set_mon_vals no callback set 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:09.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b159a700 10 monclient: dump: 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:09.675 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 10 monclient: shutdown 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 -- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 msgr2=0x7f08b4137570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 0x7f08b4137570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009e90 tx=0x7f08ac009780 comp rx=0 tx=0).stop 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 -- 172.21.15.12:0/4029530701 shutdown_connections 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f08b4099f70 0x7f08b4137570 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/4029530701 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f08b409a870 0x7f08b4099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 -- 172.21.15.12:0/4029530701 >> 172.21.15.12:0/4029530701 conn(0x7f08b409ee30 msgr2=0x7f08b409f230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 -- 172.21.15.12:0/4029530701 shutdown_connections 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.659+0000 7f08b8bd5700 1 -- 172.21.15.12:0/4029530701 wait complete. 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.660+0000 7f08b8bd5700 10 monclient: build_initial_monmap 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.660+0000 7f08b8bd5700 10 monclient: monmap: 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:09.677 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.660+0000 7f08b8bd5700 1 Processor -- start 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.660+0000 7f08b8bd5700 1 -- start start 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.660+0000 7f08b8bd5700 10 monclient: init 2023-09-29T23:14:09.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient: _add_conns ranks=[5,7,8,2,4,3,0,1,6] 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 0x7f08b4142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-h con 0x7f08b4099660 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08b4099f70 0x7f08b4142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-i con 0x7f08b4099f70 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): picked mon.noname-e con 0x7f08b409a870 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409afa0 con 0x7f08b4099660 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f08b406f3c0 con 0x7f08b4099f70 2023-09-29T23:14:09.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4090ce0 con 0x7f08b409a870 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b8bd5700 10 monclient(hunting): authenticate will time out at 1159.022329s 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:45328/0 (socket says 172.21.15.12:45328) 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 -- 172.21.15.12:0/2957415194 learned_addr learned my addr 172.21.15.12:0/2957415194 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): get_auth_request con 0x7f08b409a870 auth_method 0 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:09.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 0x7f08b4142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b1d9b700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08b4099f70 0x7f08b4142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 10 monclient(hunting): get_auth_request con 0x7f08b4099660 auth_method 0 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b1d9b700 10 monclient(hunting): get_auth_request con 0x7f08b4099f70 auth_method 0 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:09.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 -- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08b4099f70 msgr2=0x7f08b4142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08b4099f70 0x7f08b4142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 -- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 msgr2=0x7f08b4142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 0x7f08b4142630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 monclient: found mon.noname-e 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b2d9d700 1 -- 172.21.15.12:0/2957415194 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac009a20 con 0x7f08b409a870 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.661+0000 7f08b259c700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 0x7f08b4142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:09.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b2d9d700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08b409bd90 tx=0x7f08ac019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f08ac012c90 con 0x7f08b409a870 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: dump: 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:09.683 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 5 monclient: authenticate success, global_id 4346 2023-09-29T23:14:09.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 10 monclient: _renew_subs 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac012e30 con 0x7f08b409a870 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b40913a0 con 0x7f08b409a870 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f08ac026cf0 con 0x7f08b409a870 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 10 monclient: dump: 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.685 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:09.686 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b0d99700 4 set_mon_vals no callback set 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 10 monclient: _renew_subs 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b4090760 con 0x7f08b409a870 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.662+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f08ac0263f0 con 0x7f08b409a870 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08aa7fc700 1 --2- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 0x7f0894038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f08ac048e40 con 0x7f08b409a870 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b8bd5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b8bd5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b8bd5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4066f20 con 0x7f08b409a870 2023-09-29T23:14:09.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b259c700 1 --2- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 0x7f0894038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b259c700 10 monclient: get_auth_request con 0x7f0894036850 auth_method 0 2023-09-29T23:14:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.663+0000 7f08b259c700 1 --2- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 0x7f0894038d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08a40011e0 tx=0x7f08a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.668+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08ac02e950 con 0x7f08b409a870 2023-09-29T23:14:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.668+0000 7f08aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:09.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.668+0000 7f08aa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:09.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08b8bd5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08b8bd5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08b8bd5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08b4066790 con 0x7f08b409a870 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08aa7fc700 1 -- 172.21.15.12:0/2957415194 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f08ac02e360 con 0x7f08b409a870 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.901+0000 7f08aa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:09.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 msgr2=0x7f0894038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 0x7f0894038d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08a40011e0 tx=0x7f08a4009040 comp rx=0 tx=0).stop 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 10 monclient: shutdown 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 msgr2=0x7f08b408ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08b409bd90 tx=0x7f08ac019000 comp rx=0 tx=0).stop 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 shutdown_connections 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> v2:172.21.15.153:6812/83257 conn(0x7f0894036850 0x7f0894038d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f08b409a870 0x7f08b408ebc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f08b4099660 0x7f08b4142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 --2- 172.21.15.12:0/2957415194 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08b4099f70 0x7f08b4142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 >> 172.21.15.12:0/2957415194 conn(0x7f08b409ee30 msgr2=0x7f08b4096a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 shutdown_connections 2023-09-29T23:14:09.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:09.903+0000 7f08b8bd5700 1 -- 172.21.15.12:0/2957415194 wait complete. 2023-09-29T23:14:12.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:13.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.047+0000 7f3293cfd700 10 monclient: get_monmap_and_config 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.047+0000 7f3293cfd700 10 monclient: build_initial_monmap 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.047+0000 7f3293cfd700 10 monclient: monmap: 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:13.049052+0000 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:13.049052+0000 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:13.049 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:13.050 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.047+0000 7f3293cfd700 1 Processor -- start 2023-09-29T23:14:13.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 -- start start 2023-09-29T23:14:13.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient: init 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient: _add_conns ranks=[10,6,1,9,5,4,8,0,2,3,7] 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-f con 0x7f328c0a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-d con 0x7f328c0a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f328c133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c068ed0 con 0x7f328c133740 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c09e350 con 0x7f328c0a1f90 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09ef70 con 0x7f328c0a2360 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3293cfd700 10 monclient(hunting): authenticate will time out at 1162.409561s 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:52858/0 (socket says 172.21.15.12:52858) 2023-09-29T23:14:13.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 1 -- 172.21.15.12:0/3635762249 learned_addr learned my addr 172.21.15.12:0/3635762249 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): get_auth_request con 0x7f328c0a2360 auth_method 0 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 1 --2- 172.21.15.12:0/3635762249 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): get_auth_request con 0x7f328c0a1f90 auth_method 0 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:13.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.048+0000 7f3291298700 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 -- 172.21.15.12:0/3635762249 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a1f90 msgr2=0x7f328c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 --2- 172.21.15.12:0/3635762249 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 -- 172.21.15.12:0/3635762249 >> v1:172.21.15.153:6789/0 conn(0x7f328c133740 legacy=0x7f328c133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 10 monclient: _finish_hunting 0 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 monclient: found mon.noname-d 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:13.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 -- 172.21.15.12:0/3635762249 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c04c960 con 0x7f328c0a2360 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3291298700 1 --2- 172.21.15.12:0/3635762249 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f327c0099b0 tx=0x7f327c009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4239539cfa5ccce9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 1 -- 172.21.15.12:0/3635762249 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f327c009ea0 con 0x7f328c0a2360 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: dump: 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:13.062 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:13.063 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: _finish_auth 0 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 5 monclient: authenticate success, global_id 4461 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 1 -- 172.21.15.12:0/3635762249 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c00bce0 con 0x7f328c0a2360 2023-09-29T23:14:13.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 4 set_mon_vals no callback set 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 1 -- 172.21.15.12:0/3635762249 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f327c00be50 con 0x7f328c0a2360 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3290a97700 10 monclient: dump: 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:13.065 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:13.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 10 monclient: shutdown 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:13.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 -- 172.21.15.12:0/3635762249 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 msgr2=0x7f328c0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3635762249 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f327c0099b0 tx=0x7f327c009250 comp rx=0 tx=0).stop 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 -- 172.21.15.12:0/3635762249 shutdown_connections 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3635762249 >> v2:172.21.15.153:3300/0 conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3635762249 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.049+0000 7f3293cfd700 1 -- 172.21.15.12:0/3635762249 >> 172.21.15.12:0/3635762249 conn(0x7f328c0a3b50 msgr2=0x7f328c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 1 -- 172.21.15.12:0/3635762249 shutdown_connections 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 1 -- 172.21.15.12:0/3635762249 wait complete. 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 10 monclient: build_initial_monmap 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 10 monclient: monmap: 2023-09-29T23:14:13.068 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:13.069 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 1 Processor -- start 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 1 -- start start 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.050+0000 7f3293cfd700 10 monclient: init 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient: _add_conns ranks=[8,3,6,2,7,1,0,5,4] 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 0x7f328c09c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-e con 0x7f328c0a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:13.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c09c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-g con 0x7f328c0a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): picked mon.noname-d con 0x7f328c133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c1335a0 con 0x7f328c0a2360 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09ffc0 con 0x7f328c133740 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09e510 con 0x7f328c0a1f90 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3293cfd700 10 monclient(hunting): authenticate will time out at 1162.412471s 2023-09-29T23:14:13.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 0x7f328c09c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:52862/0 (socket says 172.21.15.12:52862) 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 0x7f328c09c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:45344/0 (socket says 172.21.15.12:45344) 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 -- 172.21.15.12:0/3785819339 learned_addr learned my addr 172.21.15.12:0/3785819339 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): get_auth_request con 0x7f328c133740 auth_method 0 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:13.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): get_auth_request con 0x7f328c0a1f90 auth_method 0 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c09c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): get_auth_request con 0x7f328c0a2360 auth_method 0 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:13.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291a99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f3291298700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 -- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 msgr2=0x7f328c09c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 0x7f328c09c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 -- 172.21.15.12:0/3785819339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 msgr2=0x7f328c09c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c09c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient: _finish_hunting 0 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 monclient: found mon.noname-d 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.051+0000 7f329229a700 1 -- 172.21.15.12:0/3785819339 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c0094f0 con 0x7f328c133740 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f329229a700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f328400c910 tx=0x7f3284001710 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3284012ab0 con 0x7f328c133740 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: dump: 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:13.075 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:13.076 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: _finish_auth 0 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284012c50 con 0x7f328c133740 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 5 monclient: authenticate success, global_id 4470 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 10 monclient: _renew_subs 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c12fad0 con 0x7f328c133740 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:13.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3284012dc0 con 0x7f328c133740 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3281ffb700 10 monclient: dump: 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:13.078 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 10 monclient: _renew_subs 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c1309c0 con 0x7f328c133740 2023-09-29T23:14:13.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.052+0000 7f3283fff700 4 set_mon_vals no callback set 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3284003030 con 0x7f328c133740 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3281ffb700 1 --2- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 0x7f3270038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f32840472d0 con 0x7f328c133740 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3293cfd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3293cfd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3293cfd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c070bc0 con 0x7f328c133740 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3291a99700 1 --2- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 0x7f3270038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.053+0000 7f3291a99700 10 monclient: get_auth_request con 0x7f3270036850 auth_method 0 2023-09-29T23:14:13.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.054+0000 7f3291a99700 1 --2- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 0x7f3270038d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3288009d10 tx=0x7f32880091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:13.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.058+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3284012630 con 0x7f328c133740 2023-09-29T23:14:13.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.058+0000 7f3281ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:13.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.058+0000 7f3281ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:13.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3293cfd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3293cfd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3293cfd700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f328c066790 con 0x7f328c133740 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3281ffb700 1 -- 172.21.15.12:0/3785819339 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3284028080 con 0x7f328c133740 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3281ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:13.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.294+0000 7f3281ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:13.297 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:13.297 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 msgr2=0x7f3270038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 0x7f3270038d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3288009d10 tx=0x7f32880091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 10 monclient: shutdown 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 msgr2=0x7f328c09cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f328400c910 tx=0x7f3284001710 comp rx=0 tx=0).stop 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 shutdown_connections 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c09cf30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f328c0a1f90 0x7f328c09c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> v2:172.21.15.153:6812/83257 conn(0x7f3270036850 0x7f3270038d00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 --2- 172.21.15.12:0/3785819339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c09c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 >> 172.21.15.12:0/3785819339 conn(0x7f328c0a3b50 msgr2=0x7f328c0a6190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 shutdown_connections 2023-09-29T23:14:13.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:13.296+0000 7f3293cfd700 1 -- 172.21.15.12:0/3785819339 wait complete. 2023-09-29T23:14:16.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:16.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.439+0000 7fa2cf59e700 10 monclient: get_monmap_and_config 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.439+0000 7fa2cf59e700 10 monclient: build_initial_monmap 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.439+0000 7fa2cf59e700 10 monclient: monmap: 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:16.441620+0000 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:16.441620+0000 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:16.441 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:16.442 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:16.443 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 1 Processor -- start 2023-09-29T23:14:16.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 1 -- start start 2023-09-29T23:14:16.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 10 monclient: init 2023-09-29T23:14:16.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:16.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 10 monclient: _add_conns ranks=[3,4,5,8,1,7,0,9,2,10,6] 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.440+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 0x7fa2d00a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-g con 0x7fa2d0133130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa2d0133500 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d0130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-h con 0x7fa2d012fc20 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00a27a0 con 0x7fa2d0133130 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d009adf0 con 0x7fa2d0133500 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d009bd60 con 0x7fa2d012fc20 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient(hunting): authenticate will time out at 1165.802059s 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44620/0 (socket says 172.21.15.12:44620) 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/7496261 learned_addr learned my addr 172.21.15.12:0/7496261 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): get_auth_request con 0x7fa2d0133500 auth_method 0 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:16.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 0x7fa2d00a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 10 monclient(hunting): get_auth_request con 0x7fa2d0133130 auth_method 0 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ced9d700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d0130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ced9d700 10 monclient(hunting): get_auth_request con 0x7fa2d012fc20 auth_method 0 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ced9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ced9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_done global_id 4432 payload 274 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 msgr2=0x7fa2d0130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d0130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 msgr2=0x7fa2d00a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 0x7fa2d00a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient: _finish_hunting 0 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 monclient: found mon.noname-c 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/7496261 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0068ed0 con 0x7fa2d0133500 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 0x7fa2d00a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009a10 tx=0x7fa2c4009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=72a528bba7987d3 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:16.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 1 -- 172.21.15.12:0/7496261 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa2c400bca0 con 0x7fa2d0133500 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: dump: 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:16.456 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: _finish_auth 0 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 5 monclient: authenticate success, global_id 4432 2023-09-29T23:14:16.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 1 -- 172.21.15.12:0/7496261 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c400be20 con 0x7fa2d0133500 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cf59e700 4 set_mon_vals no callback set 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 1 -- 172.21.15.12:0/7496261 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa2c40118a0 con 0x7fa2d0133500 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.441+0000 7fa2cd59a700 10 monclient: dump: 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 10 monclient: shutdown 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 -- 172.21.15.12:0/7496261 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 msgr2=0x7fa2d0133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009a10 tx=0x7fa2c4009250 comp rx=0 tx=0).stop 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 -- 172.21.15.12:0/7496261 shutdown_connections 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d0133130 0x7fa2d00a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d0130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/7496261 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fa2d0133500 0x7fa2d0133950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 -- 172.21.15.12:0/7496261 >> 172.21.15.12:0/7496261 conn(0x7fa2d00a6490 msgr2=0x7fa2d00a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 -- 172.21.15.12:0/7496261 shutdown_connections 2023-09-29T23:14:16.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 1 -- 172.21.15.12:0/7496261 wait complete. 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 10 monclient: build_initial_monmap 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.442+0000 7fa2cf59e700 10 monclient: monmap: 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:16.461 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 Processor -- start 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 -- start start 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient: init 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient: _add_conns ranks=[5,2,7,0,1,3,8,6,4] 2023-09-29T23:14:16.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d012ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-h con 0x7fa2d012fc20 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa2d0133130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa2d0133500 0x7fa2d006be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): picked mon.noname-i con 0x7fa2d0133500 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d009b020 con 0x7fa2d0133130 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0143240 con 0x7fa2d012fc20 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:16.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0137870 con 0x7fa2d0133500 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cf59e700 10 monclient(hunting): authenticate will time out at 1165.804855s 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cdd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cdd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:59900/0 (socket says 172.21.15.12:59900) 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.443+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/1949303482 learned_addr learned my addr 172.21.15.12:0/1949303482 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): get_auth_request con 0x7fa2d0133130 auth_method 0 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d012ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ced9d700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa2d0133500 0x7fa2d006be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 10 monclient(hunting): get_auth_request con 0x7fa2d012fc20 auth_method 0 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ced9d700 10 monclient(hunting): get_auth_request con 0x7fa2d0133500 auth_method 0 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ced9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:16.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ced9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa2d0133500 msgr2=0x7fa2d006be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa2d0133500 0x7fa2d006be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 msgr2=0x7fa2d012ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d012ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient: _finish_hunting 0 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 monclient: found mon.noname-b 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 -- 172.21.15.12:0/1949303482 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c40094f0 con 0x7fa2d0133130 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d012ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cdd9b700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009880 tx=0x7fa2c4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa2c40118a0 con 0x7fa2d0133130 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: dump: 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:16.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c4011a80 con 0x7fa2d0133130 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 5 monclient: authenticate success, global_id 4385 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa2c4011bf0 con 0x7fa2d0133130 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 10 monclient: _renew_subs 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d004c960 con 0x7fa2d0133130 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2ccd99700 4 set_mon_vals no callback set 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2be7fc700 10 monclient: dump: 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:16.469 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:16.470 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 10 monclient: _renew_subs 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.444+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d006d100 con 0x7fa2d0133130 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa2c4011420 con 0x7fa2d0133130 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2be7fc700 1 --2- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 0x7fa2b0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa2c4048d50 con 0x7fa2d0133130 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2cf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2cf59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:16.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2cf59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0070bc0 con 0x7fa2d0133130 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 0x7fa2b0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.445+0000 7fa2ce59c700 10 monclient: get_auth_request con 0x7fa2b0036850 auth_method 0 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.446+0000 7fa2ce59c700 1 --2- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 0x7fa2b0038d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80076e0 tx=0x7fa2c80098f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.450+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2c4015020 con 0x7fa2d0133130 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.450+0000 7fa2be7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:16.472 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.450+0000 7fa2be7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.687+0000 7fa2cf59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.687+0000 7fa2cf59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.687+0000 7fa2cf59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.687+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa2d0130460 con 0x7fa2d0133130 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.688+0000 7fa2be7fc700 1 -- 172.21.15.12:0/1949303482 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa2c40193b0 con 0x7fa2d0133130 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.688+0000 7fa2be7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:16.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.688+0000 7fa2be7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:16.690 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:16.690 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 msgr2=0x7fa2b0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 0x7fa2b0038d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80076e0 tx=0x7fa2c80098f0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 10 monclient: shutdown 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 msgr2=0x7fa2d006b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:16.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009880 tx=0x7fa2c4017000 comp rx=0 tx=0).stop 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 shutdown_connections 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa2d0133500 0x7fa2d006be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fa2d012fc20 0x7fa2d012ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> v2:172.21.15.153:6812/83257 conn(0x7fa2b0036850 0x7fa2b0038d00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 --2- 172.21.15.12:0/1949303482 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa2d0133130 0x7fa2d006b940 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.689+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 >> 172.21.15.12:0/1949303482 conn(0x7fa2d00a6490 msgr2=0x7fa2d0136130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.690+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 shutdown_connections 2023-09-29T23:14:16.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:16.690+0000 7fa2cf59e700 1 -- 172.21.15.12:0/1949303482 wait complete. 2023-09-29T23:14:19.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:19.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.832+0000 7f103f119700 10 monclient: get_monmap_and_config 2023-09-29T23:14:19.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.832+0000 7f103f119700 10 monclient: build_initial_monmap 2023-09-29T23:14:19.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.832+0000 7f103f119700 10 monclient: monmap: 2023-09-29T23:14:19.833 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:19.833 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:19.834357+0000 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:19.834357+0000 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:19.834 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:19.835 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:19.835 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:19.835 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:19.835 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:19.835 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 Processor -- start 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 -- start start 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient: init 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient: _add_conns ranks=[1,7,10,6,2,9,4,8,5,3,0] 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1038133130 addr v1:172.21.15.153:6789/0 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): picked mon.noname-i con 0x7f1038133c10 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:19.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 0x7f103813aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): picked mon.noname-f con 0x7f1038138670 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1038068ed0 con 0x7f1038133130 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103809b2a0 con 0x7f1038138670 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f10380a27a0 con 0x7f1038133c10 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.833+0000 7f103f119700 10 monclient(hunting): authenticate will time out at 1169.194821s 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 0x7f103813aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:47530/0 (socket says 172.21.15.12:47530) 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 -- 172.21.15.12:0/1296268675 learned_addr learned my addr 172.21.15.12:0/1296268675 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e117700 1 --1- 172.21.15.12:0/1296268675 >> v1:172.21.15.153:6789/0 conn(0x7f1038133130 0x7f1038133500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:56312/0 (socket says 172.21.15.12:56312) 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): get_auth_request con 0x7f1038133c10 auth_method 0 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): get_auth_request con 0x7f1038138670 auth_method 0 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:19.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 1 -- 172.21.15.12:0/1296268675 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075970058 0 0) 0x7f1038068ed0 con 0x7f1038133130 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 10 monclient(hunting): my global_id is 4438 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 1 -- 172.21.15.12:0/1296268675 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1020003570 con 0x7f1038133130 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2023-09-29T23:14:19.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 -- 172.21.15.12:0/1296268675 >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 msgr2=0x7f103813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 --2- 172.21.15.12:0/1296268675 >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 0x7f103813aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 -- 172.21.15.12:0/1296268675 >> v1:172.21.15.153:6789/0 conn(0x7f1038133130 legacy=0x7f1038133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient: _finish_hunting 0 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 monclient: found mon.noname-i 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 -- 172.21.15.12:0/1296268675 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103804c960 con 0x7f1038133c10 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103e918700 1 --2- 172.21.15.12:0/1296268675 >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 0x7f103813aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d916700 1 --2- 172.21.15.12:0/1296268675 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f102c0099e0 tx=0x7f102c009250 comp rx=0 tx=0).ready entity=mon.7 client_cookie=79ad6cc66accc9ef server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 1 -- 172.21.15.12:0/1296268675 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f102c00bac0 con 0x7f1038133c10 2023-09-29T23:14:19.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.834+0000 7f103d115700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: dump: 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:19.849 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: _finish_auth 0 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 1 -- 172.21.15.12:0/1296268675 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c00bc40 con 0x7f1038133c10 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 5 monclient: authenticate success, global_id 4399 2023-09-29T23:14:19.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 1 -- 172.21.15.12:0/1296268675 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f102c00bdb0 con 0x7f1038133c10 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 4 set_mon_vals no callback set 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103d115700 10 monclient: dump: 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:19.852 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 10 monclient: shutdown 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 -- 172.21.15.12:0/1296268675 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 msgr2=0x7f10380a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 --2- 172.21.15.12:0/1296268675 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f102c0099e0 tx=0x7f102c009250 comp rx=0 tx=0).stop 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 -- 172.21.15.12:0/1296268675 shutdown_connections 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 --2- 172.21.15.12:0/1296268675 >> v2:172.21.15.153:3300/0 conn(0x7f1038138670 0x7f103813aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 --2- 172.21.15.12:0/1296268675 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1038133c10 0x7f10380a0e80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 -- 172.21.15.12:0/1296268675 >> 172.21.15.12:0/1296268675 conn(0x7f10380a6490 msgr2=0x7f10380a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 -- 172.21.15.12:0/1296268675 shutdown_connections 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.835+0000 7f103f119700 1 -- 172.21.15.12:0/1296268675 wait complete. 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient: build_initial_monmap 2023-09-29T23:14:19.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient: monmap: 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 Processor -- start 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 -- start start 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient: init 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient: _add_conns ranks=[1,6,4,5,8,3,2,0,7] 2023-09-29T23:14:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1038133130 0x7f1038141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient(hunting): picked mon.noname-f con 0x7f1038133130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient(hunting): picked mon.noname-d con 0x7f1038133c10 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1038138670 0x7f103809ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient(hunting): picked mon.noname-c con 0x7f1038138670 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103809b3e0 con 0x7f1038133130 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.836+0000 7f103f119700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f103809b020 con 0x7f1038138670 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1038134850 con 0x7f1038133c10 2023-09-29T23:14:19.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient(hunting): authenticate will time out at 1169.198013s 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:59140/0 (socket says 172.21.15.12:59140) 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 -- 172.21.15.12:0/673221467 learned_addr learned my addr 172.21.15.12:0/673221467 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): get_auth_request con 0x7f1038133c10 auth_method 0 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1038138670 0x7f103809ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): get_auth_request con 0x7f1038138670 auth_method 0 2023-09-29T23:14:19.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1038133130 0x7f1038141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): get_auth_request con 0x7f1038133130 auth_method 0 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:19.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e918700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103e117700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 -- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1038138670 msgr2=0x7f103809ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1038138670 0x7f103809ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 -- 172.21.15.12:0/673221467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1038133130 msgr2=0x7f1038141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1038133130 0x7f1038141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient: _finish_hunting 0 2023-09-29T23:14:19.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 monclient: found mon.noname-d 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 -- 172.21.15.12:0/673221467 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028009a20 con 0x7f1038133c10 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103d916700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f102c006fd0 tx=0x7f102c017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f102c01be10 con 0x7f1038133c10 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: dump: 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:19.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:19.861 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 5 monclient: authenticate success, global_id 4479 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c025420 con 0x7f1038133c10 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f102c025a10 con 0x7f1038133c10 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient: _renew_subs 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0094f0 con 0x7f1038133c10 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:19.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103c914700 4 set_mon_vals no callback set 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f10367fc700 10 monclient: dump: 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:19.863 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient: _renew_subs 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.837+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1038134ad0 con 0x7f1038133c10 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f102c023070 con 0x7f1038133c10 2023-09-29T23:14:19.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f103f119700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f103f119700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f103f119700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f10367fc700 1 --2- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 0x7f1024038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1038067740 con 0x7f1038133c10 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.838+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f102c0484f0 con 0x7f1038133c10 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.839+0000 7f103e117700 1 --2- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 0x7f1024038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.839+0000 7f103e117700 10 monclient: get_auth_request con 0x7f1024036850 auth_method 0 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.839+0000 7f103e117700 1 --2- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 0x7f1024038d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1030007130 tx=0x7f1030000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:19.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.843+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f102c025590 con 0x7f1038133c10 2023-09-29T23:14:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.843+0000 7f10367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:19.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:19.843+0000 7f10367fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f103f119700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f103f119700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:20.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f103f119700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1038070bc0 con 0x7f1038133c10 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f10367fc700 1 -- 172.21.15.12:0/673221467 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f102c015070 con 0x7f1038133c10 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f10367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:20.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.078+0000 7f10367fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:20.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 msgr2=0x7f1024038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 0x7f1024038d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1030007130 tx=0x7f1030000bc0 comp rx=0 tx=0).stop 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 10 monclient: shutdown 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 msgr2=0x7f1038137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f102c006fd0 tx=0x7f102c017000 comp rx=0 tx=0).stop 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 shutdown_connections 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> v2:172.21.15.153:6812/83257 conn(0x7f1024036850 0x7f1024038d00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1038138670 0x7f103809ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:20.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1038133130 0x7f1038141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 --2- 172.21.15.12:0/673221467 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1038133c10 0x7f1038137790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 >> 172.21.15.12:0/673221467 conn(0x7f10380a6490 msgr2=0x7f10380a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 shutdown_connections 2023-09-29T23:14:20.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:20.080+0000 7f103f119700 1 -- 172.21.15.12:0/673221467 wait complete. 2023-09-29T23:14:23.094 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:23.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.222+0000 7f26eddc0700 10 monclient: get_monmap_and_config 2023-09-29T23:14:23.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.222+0000 7f26eddc0700 10 monclient: build_initial_monmap 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.222+0000 7f26eddc0700 10 monclient: monmap: 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:23.224314+0000 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:23.224314+0000 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:23.224 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:23.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 Processor -- start 2023-09-29T23:14:23.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 -- start start 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient: init 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient: _add_conns ranks=[10,9,6,3,0,1,4,8,7,2,5] 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f26e80a1f90 0x7f26e809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-f con 0x7f26e80a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-a con 0x7f26e80a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-d con 0x7f26e8133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e809e350 con 0x7f26e80a2360 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e809ef70 con 0x7f26e80a1f90 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f26e809da70 con 0x7f26e8133740 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26eddc0700 10 monclient(hunting): authenticate will time out at 1172.584715s 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:59146/0 (socket says 172.21.15.12:59146) 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48092/0 (socket says 172.21.15.12:48092) 2023-09-29T23:14:23.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 1 -- 172.21.15.12:0/4132414462 learned_addr learned my addr 172.21.15.12:0/4132414462 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 10 monclient(hunting): get_auth_request con 0x7f26e8133740 auth_method 0 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 10 monclient(hunting): get_auth_request con 0x7f26e80a2360 auth_method 0 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.223+0000 7f26e77fe700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.153:3300/0 conn(0x7f26e80a1f90 0x7f26e809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): get_auth_request con 0x7f26e80a1f90 auth_method 0 2023-09-29T23:14:23.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 -- 172.21.15.12:0/4132414462 >> v2:172.21.15.153:3300/0 conn(0x7f26e80a1f90 msgr2=0x7f26e809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.153:3300/0 conn(0x7f26e80a1f90 0x7f26e809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 -- 172.21.15.12:0/4132414462 >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 msgr2=0x7f26e80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 monclient: found mon.noname-d 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 -- 172.21.15.12:0/4132414462 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8068ed0 con 0x7f26e8133740 2023-09-29T23:14:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e6ffd700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e7fff700 1 --2- 172.21.15.12:0/4132414462 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26e8064b10 tx=0x7f26e0009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5ca8c6c626826e86 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 1 -- 172.21.15.12:0/4132414462 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f26e000e070 con 0x7f26e8133740 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: dump: 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:23.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:23.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26eddc0700 5 monclient: authenticate success, global_id 4488 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 1 -- 172.21.15.12:0/4132414462 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000bdb0 con 0x7f26e8133740 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26eddc0700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26eddc0700 4 set_mon_vals no callback set 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 1 -- 172.21.15.12:0/4132414462 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f26e00092e0 con 0x7f26e8133740 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.224+0000 7f26e67fc700 10 monclient: dump: 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:23.267 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:23.268 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 10 monclient: shutdown 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 -- 172.21.15.12:0/4132414462 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 msgr2=0x7f26e8133bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 --2- 172.21.15.12:0/4132414462 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26e8064b10 tx=0x7f26e0009840 comp rx=0 tx=0).stop 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 -- 172.21.15.12:0/4132414462 shutdown_connections 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.153:3300/0 conn(0x7f26e80a1f90 0x7f26e809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 --2- 172.21.15.12:0/4132414462 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26e8133740 0x7f26e8133bc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 --2- 172.21.15.12:0/4132414462 >> v2:172.21.15.12:3300/0 conn(0x7f26e80a2360 0x7f26e80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 -- 172.21.15.12:0/4132414462 >> 172.21.15.12:0/4132414462 conn(0x7f26e80a3b50 msgr2=0x7f26e80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:23.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 -- 172.21.15.12:0/4132414462 shutdown_connections 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 1 -- 172.21.15.12:0/4132414462 wait complete. 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 10 monclient: build_initial_monmap 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.225+0000 7f26eddc0700 10 monclient: monmap: 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:23.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 Processor -- start 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 -- start start 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient: init 2023-09-29T23:14:23.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient: _add_conns ranks=[1,5,4,6,2,0,7,8,3] 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e80a1f90 0x7f26e809c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-f con 0x7f26e80a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26e80a2360 0x7f26e809cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-h con 0x7f26e80a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): picked mon.noname-c con 0x7f26e8133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8130f80 con 0x7f26e80a1f90 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8130c10 con 0x7f26e8133740 2023-09-29T23:14:23.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f26e812fa40 con 0x7f26e80a2360 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26eddc0700 10 monclient(hunting): authenticate will time out at 1172.587683s 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44638/0 (socket says 172.21.15.12:44638) 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 1 -- 172.21.15.12:0/1484483185 learned_addr learned my addr 172.21.15.12:0/1484483185 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): get_auth_request con 0x7f26e8133740 auth_method 0 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e6ffd700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26e80a2360 0x7f26e809cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e77fe700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e80a1f90 0x7f26e809c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e6ffd700 10 monclient(hunting): get_auth_request con 0x7f26e80a2360 auth_method 0 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e77fe700 10 monclient(hunting): get_auth_request con 0x7f26e80a1f90 auth_method 0 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:23.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.226+0000 7f26e77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 10 monclient(hunting): handle_auth_done global_id 4450 payload 824 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 -- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26e80a2360 msgr2=0x7f26e809cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26e80a2360 0x7f26e809cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 -- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e80a1f90 msgr2=0x7f26e809c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e80a1f90 0x7f26e809c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 monclient: found mon.noname-c 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 -- 172.21.15.12:0/1484483185 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc009420 con 0x7f26e8133740 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e7fff700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26e809dbb0 tx=0x7f26e0018000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f26e0003300 con 0x7f26e8133740 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: dump: 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:23.276 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:23.277 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 5 monclient: authenticate success, global_id 4450 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00092e0 con 0x7f26e8133740 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 10 monclient: _renew_subs 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0009ae0 con 0x7f26e8133740 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f26e0007d30 con 0x7f26e8133740 2023-09-29T23:14:23.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26e5ffb700 4 set_mon_vals no callback set 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26cf7fe700 10 monclient: dump: 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 10 monclient: _renew_subs 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.227+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e804c960 con 0x7f26e8133740 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f26e0023070 con 0x7f26e8133740 2023-09-29T23:14:23.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26eddc0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26eddc0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26eddc0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e8070bc0 con 0x7f26e8133740 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26cf7fe700 1 --2- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 0x7f26d4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f26e0048ef0 con 0x7f26e8133740 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26e77fe700 1 --2- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 0x7f26d4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.228+0000 7f26e77fe700 10 monclient: get_auth_request con 0x7f26d4036850 auth_method 0 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.229+0000 7f26e77fe700 1 --2- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 0x7f26d4038d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009d10 tx=0x7f26d80091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.233+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26e00078b0 con 0x7f26e8133740 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.233+0000 7f26cf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:23.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.233+0000 7f26cf7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:23.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26eddc0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26eddc0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26eddc0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f26e8066790 con 0x7f26e8133740 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26cf7fe700 1 -- 172.21.15.12:0/1484483185 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f26e001ce00 con 0x7f26e8133740 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26cf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.473+0000 7f26cf7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:23.475 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 msgr2=0x7f26d4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 0x7f26d4038d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009d10 tx=0x7f26d80091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 10 monclient: shutdown 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 msgr2=0x7f26e812d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26e809dbb0 tx=0x7f26e0018000 comp rx=0 tx=0).stop 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 shutdown_connections 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> v2:172.21.15.153:6812/83257 conn(0x7f26d4036850 0x7f26d4038d00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e80a1f90 0x7f26e809c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f26e8133740 0x7f26e812d1d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 --2- 172.21.15.12:0/1484483185 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26e80a2360 0x7f26e809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:23.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 >> 172.21.15.12:0/1484483185 conn(0x7f26e80a3b50 msgr2=0x7f26e806f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:23.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 shutdown_connections 2023-09-29T23:14:23.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:23.475+0000 7f26eddc0700 1 -- 172.21.15.12:0/1484483185 wait complete. 2023-09-29T23:14:26.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:26.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.617+0000 7fb5cad87700 10 monclient: get_monmap_and_config 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.617+0000 7fb5cad87700 10 monclient: build_initial_monmap 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.617+0000 7fb5cad87700 10 monclient: monmap: 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:26.619485+0000 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:26.619485+0000 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:26.619 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:26.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 Processor -- start 2023-09-29T23:14:26.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 -- start start 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient: init 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient: _add_conns ranks=[1,9,6,0,10,8,7,2,5,4,3] 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb5c40a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-a con 0x7fb5c40a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-d con 0x7fb5c4138980 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c4068ed0 con 0x7fb5c40a1f90 2023-09-29T23:14:26.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c409e350 con 0x7fb5c40a2360 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c409fcf0 con 0x7fb5c4138980 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.618+0000 7fb5cad87700 10 monclient(hunting): authenticate will time out at 1175.979913s 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:48908/0 (socket says 172.21.15.12:48908) 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44042/0 (socket says 172.21.15.12:44042) 2023-09-29T23:14:26.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 -- 172.21.15.12:0/2217955103 learned_addr learned my addr 172.21.15.12:0/2217955103 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): get_auth_request con 0x7fb5c4138980 auth_method 0 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): get_auth_request con 0x7fb5c40a2360 auth_method 0 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:26.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient(hunting): handle_auth_done global_id 4497 payload 274 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 -- 172.21.15.12:0/2217955103 >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 msgr2=0x7fb5c40a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 --2- 172.21.15.12:0/2217955103 >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 -- 172.21.15.12:0/2217955103 >> v1:172.21.15.153:6789/0 conn(0x7fb5c40a1f90 legacy=0x7fb5c409f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient: _finish_hunting 0 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 monclient: found mon.noname-d 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:26.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 -- 172.21.15.12:0/2217955103 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c404c960 con 0x7fb5c4138980 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c3fff700 1 --2- 172.21.15.12:0/2217955103 >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c9324700 1 --2- 172.21.15.12:0/2217955103 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009f20 tx=0x7fb5bc009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b4ee3e1a36a849f1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 1 -- 172.21.15.12:0/2217955103 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb5bc00e070 con 0x7fb5c4138980 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: dump: 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:26.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5cad87700 5 monclient: authenticate success, global_id 4497 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 1 -- 172.21.15.12:0/2217955103 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc00be50 con 0x7fb5c4138980 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5cad87700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 1 -- 172.21.15.12:0/2217955103 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb5bc0092e0 con 0x7fb5c4138980 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5cad87700 4 set_mon_vals no callback set 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:26.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.619+0000 7fb5c37fe700 10 monclient: dump: 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:26.636 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 10 monclient: shutdown 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 -- 172.21.15.12:0/2217955103 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 msgr2=0x7fb5c413ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 --2- 172.21.15.12:0/2217955103 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009f20 tx=0x7fb5bc009840 comp rx=0 tx=0).stop 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 -- 172.21.15.12:0/2217955103 shutdown_connections 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 --2- 172.21.15.12:0/2217955103 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fb5c4138980 0x7fb5c413ad60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 --2- 172.21.15.12:0/2217955103 >> v2:172.21.15.12:3300/0 conn(0x7fb5c40a2360 0x7fb5c40a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 -- 172.21.15.12:0/2217955103 >> 172.21.15.12:0/2217955103 conn(0x7fb5c40a3b50 msgr2=0x7fb5c40a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 -- 172.21.15.12:0/2217955103 shutdown_connections 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 1 -- 172.21.15.12:0/2217955103 wait complete. 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 10 monclient: build_initial_monmap 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.620+0000 7fb5cad87700 10 monclient: monmap: 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:26.638 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 Processor -- start 2023-09-29T23:14:26.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 -- start start 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient: init 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient: _add_conns ranks=[1,7,4,2,0,5,6,3,8] 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 0x7fb5c4142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-f con 0x7fb5c40a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb5c40a2360 0x7fb5c4097a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-i con 0x7fb5c40a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): picked mon.noname-c con 0x7fb5c4138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:26.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c409ffc0 con 0x7fb5c40a1f90 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40902b0 con 0x7fb5c4138980 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c408ff40 con 0x7fb5c40a2360 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.621+0000 7fb5cad87700 10 monclient(hunting): authenticate will time out at 1175.982916s 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56812/0 (socket says 172.21.15.12:56812) 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 -- 172.21.15.12:0/429845871 learned_addr learned my addr 172.21.15.12:0/429845871 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): get_auth_request con 0x7fb5c4138980 auth_method 0 2023-09-29T23:14:26.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c3fff700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb5c40a2360 0x7fb5c4097a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c3fff700 10 monclient(hunting): get_auth_request con 0x7fb5c40a2360 auth_method 0 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c8b23700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 0x7fb5c4142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:26.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 -- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb5c40a2360 msgr2=0x7fb5c4097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb5c40a2360 0x7fb5c4097a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 -- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 msgr2=0x7fb5c4142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 0x7fb5c4142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient: _finish_hunting 0 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 monclient: found mon.noname-c 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 -- 172.21.15.12:0/429845871 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0009420 con 0x7fb5c4138980 2023-09-29T23:14:26.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c8b23700 10 monclient: get_auth_request con 0x7fb5c40a1f90 auth_method 0 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c8b23700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 0x7fb5c4142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c9324700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00b610 tx=0x7fb5bc018000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb5bc003300 con 0x7fb5c4138980 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: dump: 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.644 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: _finish_auth 0 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc0092e0 con 0x7fb5c4138980 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 5 monclient: authenticate success, global_id 4459 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 10 monclient: _renew_subs 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc009ae0 con 0x7fb5c4138980 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb5bc007000 con 0x7fb5c4138980 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:26.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c0ff9700 10 monclient: dump: 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:26.647 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5c2ffd700 4 set_mon_vals no callback set 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 10 monclient: _renew_subs 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.622+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c408fc60 con 0x7fb5c4138980 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fb5bc003050 con 0x7fb5c4138980 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5cad87700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:26.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5cad87700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5cad87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c4070bc0 con 0x7fb5c4138980 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5c0ff9700 1 --2- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 0x7fb5a8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb5bc048d00 con 0x7fb5c4138980 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5c8b23700 1 --2- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 0x7fb5a8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.623+0000 7fb5c8b23700 10 monclient: get_auth_request con 0x7fb5a8036850 auth_method 0 2023-09-29T23:14:26.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.624+0000 7fb5c8b23700 1 --2- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 0x7fb5a8038d00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8001860 tx=0x7fb5b8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.628+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5bc01cd80 con 0x7fb5c4138980 2023-09-29T23:14:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.628+0000 7fb5c0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:26.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.628+0000 7fb5c0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:26.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5cad87700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5cad87700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5cad87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5c40900b0 con 0x7fb5c4138980 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5c0ff9700 1 -- 172.21.15.12:0/429845871 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb5bc01b3d0 con 0x7fb5c4138980 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5c0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.863+0000 7fb5c0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:26.865 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 msgr2=0x7fb5a8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 0x7fb5a8038d00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8001860 tx=0x7fb5b8009040 comp rx=0 tx=0).stop 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 10 monclient: shutdown 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 msgr2=0x7fb5c408c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00b610 tx=0x7fb5bc018000 comp rx=0 tx=0).stop 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 shutdown_connections 2023-09-29T23:14:26.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> v2:172.21.15.153:6812/83257 conn(0x7fb5a8036850 0x7fb5a8038d00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c40a1f90 0x7fb5c4142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fb5c4138980 0x7fb5c408c7c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 --2- 172.21.15.12:0/429845871 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fb5c40a2360 0x7fb5c4097a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 >> 172.21.15.12:0/429845871 conn(0x7fb5c40a3b50 msgr2=0x7fb5c40a61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.865+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 shutdown_connections 2023-09-29T23:14:26.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:26.866+0000 7fb5cad87700 1 -- 172.21.15.12:0/429845871 wait complete. 2023-09-29T23:14:29.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 10 monclient: get_monmap_and_config 2023-09-29T23:14:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 10 monclient: build_initial_monmap 2023-09-29T23:14:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 10 monclient: monmap: 2023-09-29T23:14:30.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:30.007876+0000 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:30.007876+0000 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:30.008 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 1 Processor -- start 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 1 -- start start 2023-09-29T23:14:30.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.006+0000 7f617359e700 10 monclient: init 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient: _add_conns ranks=[8,9,5,6,4,0,10,2,7,3,1] 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 0x7f617406f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): picked mon.noname-e con 0x7f6174132fd0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617406c640 0x7f617412fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): picked mon.noname-a con 0x7f617406c640 addr v2:172.21.15.12:3300/0 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 0x7f6174130440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): picked mon.noname-h con 0x7f617412ffa0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6174133a70 con 0x7f617406c640 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f61741333a0 con 0x7f617412ffa0 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f617406bc60 con 0x7f6174132fd0 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617359e700 10 monclient(hunting): authenticate will time out at 1179.368305s 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617406c640 0x7f617412fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 0x7f617406f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f617406c640 0x7f617412fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44058/0 (socket says 172.21.15.12:44058) 2023-09-29T23:14:30.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 -- 172.21.15.12:0/691146003 learned_addr learned my addr 172.21.15.12:0/691146003 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): get_auth_request con 0x7f617406c640 auth_method 0 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 0x7f617406f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:52084/0 (socket says 172.21.15.12:52084) 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): get_auth_request con 0x7f6174132fd0 auth_method 0 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:30.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6172d9d700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 0x7f6174130440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient(hunting): handle_auth_done global_id 4527 payload 274 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 -- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 msgr2=0x7f617406f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 0x7f617406f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 -- 172.21.15.12:0/691146003 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 msgr2=0x7f6174130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:30.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 0x7f6174130440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 monclient: found mon.noname-a 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 -- 172.21.15.12:0/691146003 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174068ed0 con 0x7f617406c640 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6172d9d700 10 monclient: get_auth_request con 0x7f617412ffa0 auth_method 0 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6172d9d700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 0x7f6174130440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.007+0000 7f6171d9b700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617406c640 0x7f617412fa60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61680099e0 tx=0x7f6168009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4834487920342ea3 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 1 -- 172.21.15.12:0/691146003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f616800bca0 con 0x7f617406c640 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:30.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: dump: 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:30.023 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: _finish_auth 0 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 5 monclient: authenticate success, global_id 4527 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 1 -- 172.21.15.12:0/691146003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616800be00 con 0x7f617406c640 2023-09-29T23:14:30.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 4 set_mon_vals no callback set 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 1 -- 172.21.15.12:0/691146003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f61680118a0 con 0x7f617406c640 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617159a700 10 monclient: dump: 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:30.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 10 monclient: shutdown 2023-09-29T23:14:30.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 -- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617406c640 msgr2=0x7f617412fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617406c640 0x7f617412fa60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61680099e0 tx=0x7f6168009250 comp rx=0 tx=0).stop 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 -- 172.21.15.12:0/691146003 shutdown_connections 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f617412ffa0 0x7f6174130440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f6174132fd0 0x7f617406f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 --2- 172.21.15.12:0/691146003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f617406c640 0x7f617412fa60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 -- 172.21.15.12:0/691146003 >> 172.21.15.12:0/691146003 conn(0x7f6174063d00 msgr2=0x7f61740a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 -- 172.21.15.12:0/691146003 shutdown_connections 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.008+0000 7f617359e700 1 -- 172.21.15.12:0/691146003 wait complete. 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.009+0000 7f617359e700 10 monclient: build_initial_monmap 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.009+0000 7f617359e700 10 monclient: monmap: 2023-09-29T23:14:30.027 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:30.028 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.009+0000 7f617359e700 1 Processor -- start 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.009+0000 7f617359e700 1 -- start start 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.009+0000 7f617359e700 10 monclient: init 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient: _add_conns ranks=[8,6,1,2,5,7,4,0,3] 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): picked mon.noname-e con 0x7f617406c640 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): picked mon.noname-d con 0x7f617412ffa0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 0x7f617411b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): picked mon.noname-f con 0x7f6174132fd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617411e990 con 0x7f6174132fd0 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f617411eff0 con 0x7f617412ffa0 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f617411d6c0 con 0x7f617406c640 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient(hunting): authenticate will time out at 1179.371092s 2023-09-29T23:14:30.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:48922/0 (socket says 172.21.15.12:48922) 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:52086/0 (socket says 172.21.15.12:52086) 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 -- 172.21.15.12:0/29653397 learned_addr learned my addr 172.21.15.12:0/29653397 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): get_auth_request con 0x7f617412ffa0 auth_method 0 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): get_auth_request con 0x7f617406c640 auth_method 0 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:30.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 -- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 msgr2=0x7f61741423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6172d9d700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 0x7f617411b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 -- 172.21.15.12:0/29653397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 msgr2=0x7f617411b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 0x7f617411b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 monclient: found mon.noname-d 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 -- 172.21.15.12:0/29653397 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61680094f0 con 0x7f617412ffa0 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6172d9d700 10 monclient: get_auth_request con 0x7f6174132fd0 auth_method 0 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617259c700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6172d9d700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 0x7f617411b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f6171d9b700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6168000c00 tx=0x7f6168017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:30.033 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6168011ab0 con 0x7f617412ffa0 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 10 monclient: dump: 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:30.034 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168011c70 con 0x7f617412ffa0 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 5 monclient: authenticate success, global_id 4506 2023-09-29T23:14:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient: _renew_subs 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617404c960 con 0x7f617412ffa0 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.010+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6168011de0 con 0x7f617412ffa0 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f61627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f61627fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f6170d99700 4 set_mon_vals no callback set 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f61627fc700 10 monclient: dump: 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:30.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 10 monclient: _renew_subs 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617411daa0 con 0x7f617412ffa0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6168011630 con 0x7f617412ffa0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.011+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6174070bc0 con 0x7f617412ffa0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.012+0000 7f61627fc700 1 --2- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 0x7f6154038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.012+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6168048da0 con 0x7f617412ffa0 2023-09-29T23:14:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.012+0000 7f617259c700 1 --2- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 0x7f6154038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.012+0000 7f617259c700 10 monclient: get_auth_request con 0x7f6154036880 auth_method 0 2023-09-29T23:14:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.012+0000 7f617259c700 1 --2- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 0x7f6154038d30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6164011480 tx=0x7f61640091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.016+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61680118e0 con 0x7f617412ffa0 2023-09-29T23:14:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.016+0000 7f61627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.016+0000 7f61627fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f617359e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f617359e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f617359e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:30.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f617411ed50 con 0x7f617412ffa0 2023-09-29T23:14:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f61627fc700 1 -- 172.21.15.12:0/29653397 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6168015020 con 0x7f617412ffa0 2023-09-29T23:14:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f61627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:30.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.251+0000 7f61627fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:30.253 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 msgr2=0x7f6154038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 0x7f6154038d30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6164011480 tx=0x7f61640091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 10 monclient: shutdown 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 msgr2=0x7f6174142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6168000c00 tx=0x7f6168017000 comp rx=0 tx=0).stop 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 shutdown_connections 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> v2:172.21.15.153:6812/83257 conn(0x7f6154036880 0x7f6154038d30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6174132fd0 0x7f617411b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f617406c640 0x7f61741423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 --2- 172.21.15.12:0/29653397 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f617412ffa0 0x7f6174142930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 >> 172.21.15.12:0/29653397 conn(0x7f6174063d00 msgr2=0x7f6174130ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 shutdown_connections 2023-09-29T23:14:30.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:30.253+0000 7f617359e700 1 -- 172.21.15.12:0/29653397 wait complete. 2023-09-29T23:14:33.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 10 monclient: get_monmap_and_config 2023-09-29T23:14:33.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 10 monclient: build_initial_monmap 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 10 monclient: monmap: 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:33.397915+0000 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:33.397915+0000 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:33.398 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:33.399 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 1 Processor -- start 2023-09-29T23:14:33.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 1 -- start start 2023-09-29T23:14:33.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.397+0000 7f99df519700 10 monclient: init 2023-09-29T23:14:33.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient: _add_conns ranks=[10,1,5,4,6,8,2,7,3,0,9] 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99d806ef50 0x7f99d812d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): picked mon.noname-f con 0x7f99d806ef50 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f99d812d000 addr v1:172.21.15.153:6789/0 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): picked mon.noname-h con 0x7f99d81353e0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d8068ed0 con 0x7f99d812d000 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:33.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d806d1f0 con 0x7f99d806ef50 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f99d806d460 con 0x7f99d81353e0 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99df519700 10 monclient(hunting): authenticate will time out at 1182.758475s 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f99d806ef50 0x7f99d812d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:57830/0 (socket says 172.21.15.12:57830) 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 1 -- 172.21.15.12:0/3721191053 learned_addr learned my addr 172.21.15.12:0/3721191053 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): get_auth_request con 0x7f99d81353e0 auth_method 0 2023-09-29T23:14:33.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dcab4700 1 --1- 172.21.15.12:0/3721191053 >> v1:172.21.15.153:6789/0 conn(0x7f99d812d000 0x7f99d806f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:39242/0 (socket says 172.21.15.12:39242) 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): get_auth_request con 0x7f99d806ef50 auth_method 0 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:33.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.398+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_done global_id 4474 payload 274 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 -- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 msgr2=0x7f99d81377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 -- 172.21.15.12:0/3721191053 >> v1:172.21.15.153:6789/0 conn(0x7f99d812d000 legacy=0x7f99d806f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 10 monclient: _finish_hunting 0 2023-09-29T23:14:33.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 monclient: found mon.noname-f 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 -- 172.21.15.12:0/3721191053 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d804c960 con 0x7f99d806ef50 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 1 -- 172.21.15.12:0/3721191053 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967916558 0 0) 0x7f99d8068ed0 con 0x7f99d812d000 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99ddab6700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99dd2b5700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d806ef50 0x7f99d812d630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99d40098d0 tx=0x7f99d4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b8207967d954b8 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 1 -- 172.21.15.12:0/3721191053 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f99d4005c10 con 0x7f99d806ef50 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:33.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: dump: 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:33.415 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: _finish_auth 0 2023-09-29T23:14:33.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 1 -- 172.21.15.12:0/3721191053 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d4005d70 con 0x7f99d806ef50 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 1 -- 172.21.15.12:0/3721191053 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f99d400b430 con 0x7f99d806ef50 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99cffff700 10 monclient: dump: 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.417 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:33.418 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99df519700 5 monclient: authenticate success, global_id 4474 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99df519700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.399+0000 7f99df519700 4 set_mon_vals no callback set 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 10 monclient: shutdown 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:33.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 -- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d806ef50 msgr2=0x7f99d812d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d806ef50 0x7f99d812d630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99d40098d0 tx=0x7f99d4009120 comp rx=0 tx=0).stop 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 -- 172.21.15.12:0/3721191053 shutdown_connections 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f99d81353e0 0x7f99d81377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 --2- 172.21.15.12:0/3721191053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d806ef50 0x7f99d812d630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 -- 172.21.15.12:0/3721191053 >> 172.21.15.12:0/3721191053 conn(0x7f99d80a3330 msgr2=0x7f99d80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 -- 172.21.15.12:0/3721191053 shutdown_connections 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 1 -- 172.21.15.12:0/3721191053 wait complete. 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 10 monclient: build_initial_monmap 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.400+0000 7f99df519700 10 monclient: monmap: 2023-09-29T23:14:33.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:33.421 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 Processor -- start 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 -- start start 2023-09-29T23:14:33.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient: init 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient: _add_conns ranks=[3,1,4,0,6,5,7,8,2] 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d806ef50 0x7f99d8129840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): picked mon.noname-g con 0x7f99d806ef50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d812d000 0x7f99d8128900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): picked mon.noname-f con 0x7f99d812d000 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): picked mon.noname-c con 0x7f99d81353e0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:33.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d8126eb0 con 0x7f99d812d000 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d809f790 con 0x7f99d806ef50 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f99d8127c40 con 0x7f99d81353e0 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99df519700 10 monclient(hunting): authenticate will time out at 1182.761842s 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:55206/0 (socket says 172.21.15.12:55206) 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 1 -- 172.21.15.12:0/1039545521 learned_addr learned my addr 172.21.15.12:0/1039545521 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:33.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 10 monclient(hunting): get_auth_request con 0x7f99d81353e0 auth_method 0 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.401+0000 7f99ddab6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d806ef50 0x7f99d8129840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d812d000 0x7f99d8128900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): get_auth_request con 0x7f99d806ef50 auth_method 0 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:33.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): get_auth_request con 0x7f99d812d000 auth_method 0 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:33.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dd2b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99dcab4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 -- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d806ef50 msgr2=0x7f99d8129840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d806ef50 0x7f99d8129840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 -- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d812d000 msgr2=0x7f99d8128900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d812d000 0x7f99d8128900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient: _finish_hunting 0 2023-09-29T23:14:33.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 monclient: found mon.noname-c 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 -- 172.21.15.12:0/1039545521 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0009ae0 con 0x7f99d81353e0 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99ddab6700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009f10 tx=0x7f99d000b040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f99d000cde0 con 0x7f99d81353e0 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: dump: 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:33.428 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:33.429 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: _finish_auth 0 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:33.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 5 monclient: authenticate success, global_id 4468 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000f040 con 0x7f99d81353e0 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f99d0014900 con 0x7f99d81353e0 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 10 monclient: _renew_subs 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d40093c0 con 0x7f99d81353e0 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cf7fe700 4 set_mon_vals no callback set 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:33.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99cd7fa700 10 monclient: dump: 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 10 monclient: _renew_subs 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.402+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d8098e60 con 0x7f99d81353e0 2023-09-29T23:14:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f99d0014480 con 0x7f99d81353e0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99df519700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99df519700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99df519700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99cd7fa700 1 --2- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 0x7f99bc038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d8128230 con 0x7f99d81353e0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f99d00477d0 con 0x7f99d81353e0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99dd2b5700 1 --2- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 0x7f99bc038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.403+0000 7f99dd2b5700 10 monclient: get_auth_request con 0x7f99bc036780 auth_method 0 2023-09-29T23:14:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.404+0000 7f99dd2b5700 1 --2- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 0x7f99bc038c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009390 tx=0x7f99d4016000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.408+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99d001d410 con 0x7f99d81353e0 2023-09-29T23:14:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.408+0000 7f99cd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.408+0000 7f99cd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99df519700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99df519700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:33.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99df519700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f99d8070bc0 con 0x7f99d81353e0 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99cd7fa700 1 -- 172.21.15.12:0/1039545521 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f99d0037030 con 0x7f99d81353e0 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99cd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.645+0000 7f99cd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:33.646 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 msgr2=0x7f99bc038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 0x7f99bc038c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009390 tx=0x7f99d4016000 comp rx=0 tx=0).stop 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 10 monclient: shutdown 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 msgr2=0x7f99d80979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009f10 tx=0x7f99d000b040 comp rx=0 tx=0).stop 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 shutdown_connections 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f99d81353e0 0x7f99d80979d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> v2:172.21.15.153:6812/83257 conn(0x7f99bc036780 0x7f99bc038c30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d806ef50 0x7f99d8129840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 --2- 172.21.15.12:0/1039545521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d812d000 0x7f99d8128900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 >> 172.21.15.12:0/1039545521 conn(0x7f99d80a3330 msgr2=0x7f99d812c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 shutdown_connections 2023-09-29T23:14:33.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:33.647+0000 7f99df519700 1 -- 172.21.15.12:0/1039545521 wait complete. 2023-09-29T23:14:36.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:36.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.788+0000 7f255ef85700 10 monclient: get_monmap_and_config 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.788+0000 7f255ef85700 10 monclient: build_initial_monmap 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.788+0000 7f255ef85700 10 monclient: monmap: 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:36.789425+0000 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:36.789425+0000 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:36.789 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:36.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 1 Processor -- start 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 1 -- start start 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient: init 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient: _add_conns ranks=[4,5,0,9,2,10,8,3,1,6,7] 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-c con 0x7f25580a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 0x7f25580a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-h con 0x7f25580a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.789+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2558133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2558068ed0 con 0x7f2558133740 2023-09-29T23:14:36.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f255809e350 con 0x7f25580a1f90 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f255809ef70 con 0x7f25580a2360 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 10 monclient(hunting): authenticate will time out at 1186.150071s 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:55214/0 (socket says 172.21.15.12:55214) 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 -- 172.21.15.12:0/3528703898 learned_addr learned my addr 172.21.15.12:0/3528703898 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): get_auth_request con 0x7f25580a1f90 auth_method 0 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:36.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255d522700 1 --1- 172.21.15.12:0/3528703898 >> v1:172.21.15.12:6789/0 conn(0x7f2558133740 0x7f2558133bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38696/0 (socket says 172.21.15.12:38696) 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 0x7f25580a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 10 monclient(hunting): get_auth_request con 0x7f25580a2360 auth_method 0 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient(hunting): handle_auth_done global_id 4477 payload 274 2023-09-29T23:14:36.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 -- 172.21.15.12:0/3528703898 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 msgr2=0x7f25580a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 0x7f25580a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 -- 172.21.15.12:0/3528703898 >> v1:172.21.15.12:6789/0 conn(0x7f2558133740 legacy=0x7f2558133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient: _finish_hunting 0 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 monclient: found mon.noname-c 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 1 -- 172.21.15.12:0/3528703898 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194283859 0 0) 0x7f2558068ed0 con 0x7f2558133740 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 -- 172.21.15.12:0/3528703898 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255804c960 con 0x7f25580a1f90 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:14:36.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f2557fff700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 0x7f25580a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255cd21700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a10 tx=0x7f2540009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=5c699957bfffc0d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 1 -- 172.21.15.12:0/3528703898 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f254000bca0 con 0x7f25580a1f90 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: dump: 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:36.804 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:36.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 5 monclient: authenticate success, global_id 4477 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 1 -- 172.21.15.12:0/3528703898 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254000be20 con 0x7f25580a1f90 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f255ef85700 4 set_mon_vals no callback set 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 1 -- 172.21.15.12:0/3528703898 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f25400118a0 con 0x7f25580a1f90 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.790+0000 7f25577fe700 10 monclient: dump: 2023-09-29T23:14:36.806 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:36.807 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 10 monclient: shutdown 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 -- 172.21.15.12:0/3528703898 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 msgr2=0x7f255809f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a10 tx=0x7f2540009250 comp rx=0 tx=0).stop 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 -- 172.21.15.12:0/3528703898 shutdown_connections 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f25580a1f90 0x7f255809f5e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:36.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 --2- 172.21.15.12:0/3528703898 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f25580a2360 0x7f25580a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 -- 172.21.15.12:0/3528703898 >> 172.21.15.12:0/3528703898 conn(0x7f25580a3b50 msgr2=0x7f25580a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 -- 172.21.15.12:0/3528703898 shutdown_connections 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 1 -- 172.21.15.12:0/3528703898 wait complete. 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.791+0000 7f255ef85700 10 monclient: build_initial_monmap 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.792+0000 7f255ef85700 10 monclient: monmap: 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:36.809 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.792+0000 7f255ef85700 1 Processor -- start 2023-09-29T23:14:36.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.792+0000 7f255ef85700 1 -- start start 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.792+0000 7f255ef85700 10 monclient: init 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.792+0000 7f255ef85700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient: _add_conns ranks=[3,6,5,2,0,7,1,8,4] 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25580a1f90 0x7f255809c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-g con 0x7f25580a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-d con 0x7f25580a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2558133740 0x7f255812d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): picked mon.noname-h con 0x7f2558133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2558130a10 con 0x7f25580a1f90 2023-09-29T23:14:36.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f255812f4d0 con 0x7f2558133740 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f255809ffc0 con 0x7f25580a2360 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient(hunting): authenticate will time out at 1186.153091s 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:45160/0 (socket says 172.21.15.12:45160) 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 -- 172.21.15.12:0/1625083295 learned_addr learned my addr 172.21.15.12:0/1625083295 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): get_auth_request con 0x7f25580a2360 auth_method 0 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:36.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2558133740 0x7f255812d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): get_auth_request con 0x7f2558133740 auth_method 0 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25580a1f90 0x7f255809c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): get_auth_request con 0x7f25580a1f90 auth_method 0 2023-09-29T23:14:36.813 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255d522700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255cd21700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient(hunting): handle_auth_done global_id 4515 payload 824 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 -- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2558133740 msgr2=0x7f255812d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:36.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2558133740 0x7f255812d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 -- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25580a1f90 msgr2=0x7f255809c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25580a1f90 0x7f255809c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 monclient: found mon.noname-d 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 -- 172.21.15.12:0/1625083295 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400094f0 con 0x7f25580a2360 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2557fff700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f254800e1e0 tx=0x7f25480099e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2548009a50 con 0x7f25580a2360 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:14:36.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: dump: 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:36.816 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: _finish_auth 0 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 5 monclient: authenticate success, global_id 4515 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254800ecc0 con 0x7f25580a2360 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f254800ee30 con 0x7f25580a2360 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient: _renew_subs 2023-09-29T23:14:36.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255809e510 con 0x7f25580a2360 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2556ffd700 4 set_mon_vals no callback set 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.793+0000 7f2554ff9700 10 monclient: dump: 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:36.818 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.794+0000 7f255ef85700 10 monclient: _renew_subs 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.794+0000 7f255ef85700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:36.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.794+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255812eee0 con 0x7f25580a2360 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.794+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f254801e460 con 0x7f25580a2360 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255ef85700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255ef85700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255ef85700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2558070bc0 con 0x7f25580a2360 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f2554ff9700 1 --2- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 0x7f2538038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2548047950 con 0x7f25580a2360 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255cd21700 1 --2- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 0x7f2538038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255cd21700 10 monclient: get_auth_request con 0x7f2538036850 auth_method 0 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.795+0000 7f255cd21700 1 --2- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 0x7f2538038d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a10 tx=0x7f2540017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:36.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.799+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f254801b6d0 con 0x7f25580a2360 2023-09-29T23:14:36.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.799+0000 7f2554ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:36.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:36.799+0000 7f2554ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f255ef85700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:37.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f255ef85700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f255ef85700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2558066790 con 0x7f25580a2360 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f2554ff9700 1 -- 172.21.15.12:0/1625083295 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f254801b310 con 0x7f25580a2360 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f2554ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.034+0000 7f2554ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:37.035 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:37.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 msgr2=0x7f2538038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 0x7f2538038d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a10 tx=0x7f2540017000 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 10 monclient: shutdown 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 msgr2=0x7f255809cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f254800e1e0 tx=0x7f25480099e0 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 shutdown_connections 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2558133740 0x7f255812d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> v2:172.21.15.153:6812/83257 conn(0x7f2538036850 0x7f2538038d00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25580a1f90 0x7f255809c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 --2- 172.21.15.12:0/1625083295 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f25580a2360 0x7f255809cbb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:37.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 >> 172.21.15.12:0/1625083295 conn(0x7f25580a3b50 msgr2=0x7f255806dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 shutdown_connections 2023-09-29T23:14:37.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:37.036+0000 7f255ef85700 1 -- 172.21.15.12:0/1625083295 wait complete. 2023-09-29T23:14:40.049 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:40.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.180+0000 7f61350fb700 10 monclient: get_monmap_and_config 2023-09-29T23:14:40.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.180+0000 7f61350fb700 10 monclient: build_initial_monmap 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.180+0000 7f61350fb700 10 monclient: monmap: 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:40.181437+0000 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:40.181437+0000 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:40.181 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:40.182 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 Processor -- start 2023-09-29T23:14:40.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 -- start start 2023-09-29T23:14:40.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient: init 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient: _add_conns ranks=[0,3,4,6,1,10,2,9,7,5,8] 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61300a1f90 addr v1:172.21.15.12:6789/0 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61300a2360 0x7f61300a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-g con 0x7f61300a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 0x7f613013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-c con 0x7f6130138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6130068ed0 con 0x7f61300a1f90 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613009e350 con 0x7f61300a2360 2023-09-29T23:14:40.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f613009fcf0 con 0x7f6130138980 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f61350fb700 10 monclient(hunting): authenticate will time out at 1189.541750s 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612ed9d700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f61300a1f90 0x7f613009f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:38710/0 (socket says 172.21.15.12:38710) 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612f59e700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 0x7f613013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612ed9d700 1 -- 172.21.15.12:0/3306802525 learned_addr learned my addr 172.21.15.12:0/3306802525 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612f59e700 10 monclient(hunting): get_auth_request con 0x7f6130138980 auth_method 0 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.181+0000 7f612f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:40.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61300a2360 0x7f61300a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): get_auth_request con 0x7f61300a2360 auth_method 0 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient(hunting): handle_auth_done global_id 4486 payload 274 2023-09-29T23:14:40.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 1 -- 172.21.15.12:0/3306802525 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568526067 0 0) 0x7f6130068ed0 con 0x7f61300a1f90 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 -- 172.21.15.12:0/3306802525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61300a2360 msgr2=0x7f61300a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61300a2360 0x7f61300a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 -- 172.21.15.12:0/3306802525 >> v1:172.21.15.12:6789/0 conn(0x7f61300a1f90 legacy=0x7f613009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient: _finish_hunting 0 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 monclient: found mon.noname-c 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 -- 172.21.15.12:0/3306802525 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613004c960 con 0x7f6130138980 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612f59e700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 0x7f613013ad60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6128009e90 tx=0x7f6128009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=137edf468eedf387 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:40.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 1 -- 172.21.15.12:0/3306802525 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f612800e070 con 0x7f6130138980 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: dump: 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:40.195 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: _finish_auth 0 2023-09-29T23:14:40.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 1 -- 172.21.15.12:0/3306802525 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612800bc30 con 0x7f6130138980 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f61350fb700 5 monclient: authenticate success, global_id 4486 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f61350fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f61350fb700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 1 -- 172.21.15.12:0/3306802525 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f612800bda0 con 0x7f6130138980 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f61350fb700 4 set_mon_vals no callback set 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.182+0000 7f612dd9b700 10 monclient: dump: 2023-09-29T23:14:40.197 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:40.198 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 10 monclient: shutdown 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 -- 172.21.15.12:0/3306802525 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 msgr2=0x7f613013ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 0x7f613013ad60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6128009e90 tx=0x7f6128009780 comp rx=0 tx=0).stop 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 -- 172.21.15.12:0/3306802525 shutdown_connections 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6130138980 0x7f613013ad60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 --2- 172.21.15.12:0/3306802525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61300a2360 0x7f61300a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 -- 172.21.15.12:0/3306802525 >> 172.21.15.12:0/3306802525 conn(0x7f61300a3b50 msgr2=0x7f61300a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 -- 172.21.15.12:0/3306802525 shutdown_connections 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 1 -- 172.21.15.12:0/3306802525 wait complete. 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 10 monclient: build_initial_monmap 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.183+0000 7f61350fb700 10 monclient: monmap: 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:40.200 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 Processor -- start 2023-09-29T23:14:40.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 -- start start 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient: init 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient: _add_conns ranks=[5,7,2,3,8,0,1,4,6] 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 0x7f6130141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-h con 0x7f61300a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 0x7f6130137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-i con 0x7f61300a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): picked mon.noname-b con 0x7f6130138980 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:40.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613012f200 con 0x7f6130138980 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6130130c30 con 0x7f61300a1f90 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f613009e510 con 0x7f61300a2360 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f61350fb700 10 monclient(hunting): authenticate will time out at 1189.544756s 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:51476/0 (socket says 172.21.15.12:51476) 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 1 -- 172.21.15.12:0/1679162995 learned_addr learned my addr 172.21.15.12:0/1679162995 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): get_auth_request con 0x7f6130138980 auth_method 0 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:40.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.184+0000 7f612f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 0x7f6130137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 0x7f6130141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 10 monclient(hunting): get_auth_request con 0x7f61300a2360 auth_method 0 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:40.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 10 monclient(hunting): get_auth_request con 0x7f61300a1f90 auth_method 0 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 -- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 msgr2=0x7f6130137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 0x7f6130137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 -- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 msgr2=0x7f6130141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 0x7f6130141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 10 monclient: _finish_hunting 0 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 monclient: found mon.noname-b 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:40.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 -- 172.21.15.12:0/1679162995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6128009a20 con 0x7f6130138980 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612e59c700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 0x7f6130137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612ed9d700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 0x7f6130141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612f59e700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f61300a0bc0 tx=0x7f6128019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6128012cd0 con 0x7f6130138980 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: dump: 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:40.206 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:40.207 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: _finish_auth 0 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 5 monclient: authenticate success, global_id 4394 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6128012e70 con 0x7f6130138980 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 10 monclient: _renew_subs 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6130130f10 con 0x7f6130138980 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6128026cf0 con 0x7f6130138980 2023-09-29T23:14:40.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f612d59a700 4 set_mon_vals no callback set 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f6116ffd700 10 monclient: dump: 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:40.209 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 10 monclient: _renew_subs 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.185+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6130142cd0 con 0x7f6130138980 2023-09-29T23:14:40.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f61280263f0 con 0x7f6130138980 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f61350fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f61350fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f61350fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61300a0030 con 0x7f6130138980 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f6116ffd700 1 --2- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 0x7f611c038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6128048d50 con 0x7f6130138980 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f612ed9d700 1 --2- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 0x7f611c038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.186+0000 7f612ed9d700 10 monclient: get_auth_request con 0x7f611c036780 auth_method 0 2023-09-29T23:14:40.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.187+0000 7f612ed9d700 1 --2- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 0x7f611c038c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61200011e0 tx=0x7f6120009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:40.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.191+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61300a0030 con 0x7f6130138980 2023-09-29T23:14:40.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.191+0000 7f6116ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:40.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.191+0000 7f6116ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:40.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f61350fb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:40.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f61350fb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:40.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f61350fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:40.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61300a0030 con 0x7f6130138980 2023-09-29T23:14:40.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f6116ffd700 1 -- 172.21.15.12:0/1679162995 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f61300a0030 con 0x7f6130138980 2023-09-29T23:14:40.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f6116ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:40.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.431+0000 7f6116ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:40.432 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:40.433 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 msgr2=0x7f611c038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 0x7f611c038c30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61200011e0 tx=0x7f6120009040 comp rx=0 tx=0).stop 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 10 monclient: shutdown 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 msgr2=0x7f613012d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f61300a0bc0 tx=0x7f6128019000 comp rx=0 tx=0).stop 2023-09-29T23:14:40.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 shutdown_connections 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> v2:172.21.15.153:6812/83257 conn(0x7f611c036780 0x7f611c038c30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6130138980 0x7f613012d560 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f61300a1f90 0x7f6130141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 --2- 172.21.15.12:0/1679162995 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f61300a2360 0x7f6130137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 >> 172.21.15.12:0/1679162995 conn(0x7f61300a3b50 msgr2=0x7f61300a5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 shutdown_connections 2023-09-29T23:14:40.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:40.433+0000 7f61350fb700 1 -- 172.21.15.12:0/1679162995 wait complete. 2023-09-29T23:14:43.446 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:43.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.573+0000 7f796c8ed700 10 monclient: get_monmap_and_config 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.573+0000 7f796c8ed700 10 monclient: build_initial_monmap 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.573+0000 7f796c8ed700 10 monclient: monmap: 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:43.574588+0000 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:43.574588+0000 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.574 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:43.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.574+0000 7f796c8ed700 1 Processor -- start 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.574+0000 7f796c8ed700 1 -- start start 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.574+0000 7f796c8ed700 10 monclient: init 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient: _add_conns ranks=[0,10,5,3,2,1,8,9,7,6,4] 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7968133130 addr v1:172.21.15.12:6789/0 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7968133c10 0x7f79680a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-f con 0x7f7968133c10 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7968138670 0x7f796813aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-h con 0x7f7968138670 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7968068ed0 con 0x7f7968133130 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796809b2a0 con 0x7f7968133c10 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a27a0 con 0x7f7968138670 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 10 monclient(hunting): authenticate will time out at 1192.935197s 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79677fe700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7968133130 0x7f7968133500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33080/0 (socket says 172.21.15.12:33080) 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79677fe700 1 -- 172.21.15.12:0/1499080394 learned_addr learned my addr 172.21.15.12:0/1499080394 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7968138670 0x7f796813aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 1 --2- 172.21.15.12:0/1499080394 >> v2:172.21.15.153:3300/0 conn(0x7f7968133c10 0x7f79680a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): get_auth_request con 0x7f7968138670 auth_method 0 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): get_auth_request con 0x7f7968133c10 auth_method 0 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565385801 0 0) 0x7f7968068ed0 con 0x7f7968133130 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): my global_id is 4554 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7950003570 con 0x7f7968133130 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621492933 0 0) 0x7f7950003570 con 0x7f7968133130 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7968138670 msgr2=0x7f796813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:43.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7968138670 0x7f796813aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 >> v2:172.21.15.153:3300/0 conn(0x7f7968133c10 msgr2=0x7f79680a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 --2- 172.21.15.12:0/1499080394 >> v2:172.21.15.153:3300/0 conn(0x7f7968133c10 0x7f79680a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: _finish_hunting 0 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796804c960 con 0x7f7968133130 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f79580018f0 con 0x7f7968133130 2023-09-29T23:14:43.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 5 monclient: authenticate success, global_id 4554 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: dump: 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:43.592 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:43.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 >> v1:172.21.15.12:6789/0 conn(0x7f7968133130 legacy=0x7f7968133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient: _add_conns ranks=[3,7,4,5,8,2,0,6,1] 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79500042b0 0x7f7950004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): picked mon.g con 0x7f79500042b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7950004da0 0x7f7950005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): picked mon.i con 0x7f7950004da0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 0x7f795000bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): picked mon.c con 0x7f7950009b40 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796809b2a0 con 0x7f79500042b0 2023-09-29T23:14:43.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a27a0 con 0x7f7950009b40 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f795000c8b0 con 0x7f7950004da0 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f79667fc700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.575+0000 7f796c8ed700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 0x7f795000bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): get_auth_request con 0x7f7950009b40 auth_method 0 2023-09-29T23:14:43.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79500042b0 0x7f7950004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): get_auth_request con 0x7f79500042b0 auth_method 0 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7950004da0 0x7f7950005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): get_auth_request con 0x7f7950004da0 auth_method 0 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:14:43.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient(hunting): handle_auth_done global_id 4554 payload 293 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 -- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7950004da0 msgr2=0x7f7950005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f7950004da0 0x7f7950005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 -- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79500042b0 msgr2=0x7f7950004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79500042b0 0x7f7950004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 monclient: found mon.c 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 -- 172.21.15.12:0/1499080394 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7950003570 con 0x7f7950009b40 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 10 monclient: _finish_auth 0 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:43.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 -- 172.21.15.12:0/1499080394 reap_dead start 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f7967fff700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 0x7f795000bfa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f796000e210 tx=0x7f796000b040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=da987fa3b969421f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7960014480 con 0x7f7950009b40 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: dump: 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.599 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:43.600 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f796c8ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796000f040 con 0x7f7950009b40 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f796c8ed700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f796c8ed700 4 set_mon_vals no callback set 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 1 -- 172.21.15.12:0/1499080394 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7960014ac0 con 0x7f7950009b40 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:43.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.576+0000 7f79667fc700 10 monclient: dump: 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:43.602 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 10 monclient: shutdown 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 -- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 msgr2=0x7f795000bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 0x7f795000bfa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f796000e210 tx=0x7f796000b040 comp rx=0 tx=0).stop 2023-09-29T23:14:43.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 -- 172.21.15.12:0/1499080394 shutdown_connections 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 --2- 172.21.15.12:0/1499080394 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7950009b40 0x7f795000bfa0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 -- 172.21.15.12:0/1499080394 >> 172.21.15.12:0/1499080394 conn(0x7f79680a6490 msgr2=0x7f79680a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 -- 172.21.15.12:0/1499080394 shutdown_connections 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 1 -- 172.21.15.12:0/1499080394 wait complete. 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 10 monclient: build_initial_monmap 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.577+0000 7f796c8ed700 10 monclient: monmap: 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:43.604 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:43.605 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 Processor -- start 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 -- start start 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient: init 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient: _add_conns ranks=[1,8,2,7,3,4,6,0,5] 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7950009b40 0x7f7968137b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-f con 0x7f7950009b40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-e con 0x7f7968134d40 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f796809f320 0x7f796809f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): picked mon.noname-b con 0x7f796809f320 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7968135f10 con 0x7f7950009b40 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796809fcb0 con 0x7f796809f320 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f796809b3e0 con 0x7f7968134d40 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f796c8ed700 10 monclient(hunting): authenticate will time out at 1192.938826s 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56402/0 (socket says 172.21.15.12:56402) 2023-09-29T23:14:43.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 1 -- 172.21.15.12:0/605071057 learned_addr learned my addr 172.21.15.12:0/605071057 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 10 monclient(hunting): get_auth_request con 0x7f7968134d40 auth_method 0 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7966ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7967fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f796809f320 0x7f796809f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7967fff700 10 monclient(hunting): get_auth_request con 0x7f796809f320 auth_method 0 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7967fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7967fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.578+0000 7f7967fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7950009b40 0x7f7968137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): get_auth_request con 0x7f7950009b40 auth_method 0 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7967fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:43.609 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f79677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 -- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f796809f320 msgr2=0x7f796809f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f796809f320 0x7f796809f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 -- 172.21.15.12:0/605071057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7950009b40 msgr2=0x7f7968137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7950009b40 0x7f7968137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 monclient: found mon.noname-e 2023-09-29T23:14:43.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 -- 172.21.15.12:0/605071057 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796809b020 con 0x7f7968134d40 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7966ffd700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f795c009170 tx=0x7f795c0091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f795c00fd90 con 0x7f7968134d40 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: dump: 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.611 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:43.612 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c014440 con 0x7f7968134d40 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 5 monclient: authenticate success, global_id 4382 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 10 monclient: _renew_subs 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7968135c30 con 0x7f7968134d40 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f795c014a30 con 0x7f7968134d40 2023-09-29T23:14:43.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f794f7fe700 10 monclient: dump: 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:43.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f7965ffb700 4 set_mon_vals no callback set 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 10 monclient: _renew_subs 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:43.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.579+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968137080 con 0x7f7968134d40 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f795c014b90 con 0x7f7968134d40 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f796c8ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f796c8ed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f796c8ed700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968067740 con 0x7f7968134d40 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f794f7fe700 1 --2- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 0x7f7954038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f795c018030 con 0x7f7968134d40 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f79677fe700 1 --2- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 0x7f7954038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.580+0000 7f79677fe700 10 monclient: get_auth_request con 0x7f7954036850 auth_method 0 2023-09-29T23:14:43.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.581+0000 7f79677fe700 1 --2- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 0x7f7954038d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7958001a60 tx=0x7f796809c600 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.585+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f795c014e40 con 0x7f7968134d40 2023-09-29T23:14:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.585+0000 7f794f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:43.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.585+0000 7f794f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:43.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.820+0000 7f796c8ed700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.820+0000 7f796c8ed700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.820+0000 7f796c8ed700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.820+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7968136150 con 0x7f7968134d40 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.821+0000 7f794f7fe700 1 -- 172.21.15.12:0/605071057 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f795c01baf0 con 0x7f7968134d40 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.821+0000 7f794f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.821+0000 7f794f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:43.822 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:43.823 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 msgr2=0x7f7954038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 0x7f7954038d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7958001a60 tx=0x7f796809c600 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 10 monclient: shutdown 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 msgr2=0x7f796809ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f795c009170 tx=0x7f795c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 shutdown_connections 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> v2:172.21.15.153:6812/83257 conn(0x7f7954036850 0x7f7954038d00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f796809f320 0x7f796809f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7950009b40 0x7f7968137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 --2- 172.21.15.12:0/605071057 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7968134d40 0x7f796809ede0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:43.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 >> 172.21.15.12:0/605071057 conn(0x7f79680a6490 msgr2=0x7f79680a2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:43.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 shutdown_connections 2023-09-29T23:14:43.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:43.823+0000 7f796c8ed700 1 -- 172.21.15.12:0/605071057 wait complete. 2023-09-29T23:14:46.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.966+0000 7f366f59e700 10 monclient: get_monmap_and_config 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.966+0000 7f366f59e700 10 monclient: build_initial_monmap 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.966+0000 7f366f59e700 10 monclient: monmap: 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:46.967601+0000 2023-09-29T23:14:46.967 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:46.967601+0000 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:46.968 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:46.969 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:46.969 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.967+0000 7f366f59e700 1 Processor -- start 2023-09-29T23:14:46.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.967+0000 7f366f59e700 1 -- start start 2023-09-29T23:14:46.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.967+0000 7f366f59e700 10 monclient: init 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient: _add_conns ranks=[6,1,8,4,10,2,9,7,3,0,5] 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-d con 0x7f3670133130 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3670133500 addr v1:172.21.15.153:6789/0 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-e con 0x7f3670138670 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3670068ed0 con 0x7f3670133500 2023-09-29T23:14:46.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f367009b2a0 con 0x7f3670133130 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f36700a27a0 con 0x7f3670138670 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366f59e700 10 monclient(hunting): authenticate will time out at 1196.328114s 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56404/0 (socket says 172.21.15.12:56404) 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35730/0 (socket says 172.21.15.12:35730) 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 -- 172.21.15.12:0/1746530918 learned_addr learned my addr 172.21.15.12:0/1746530918 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): get_auth_request con 0x7f3670138670 auth_method 0 2023-09-29T23:14:46.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): get_auth_request con 0x7f3670133130 auth_method 0 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:46.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 -- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 msgr2=0x7f36700a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 -- 172.21.15.12:0/1746530918 >> v1:172.21.15.153:6789/0 conn(0x7f3670133500 legacy=0x7f3670133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 monclient: found mon.noname-e 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 -- 172.21.15.12:0/1746530918 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367004c960 con 0x7f3670138670 2023-09-29T23:14:46.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366e59c700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366ed9d700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3668009f50 tx=0x7f3668009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=49d57225911d81ac server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366d59a700 1 -- 172.21.15.12:0/1746530918 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f366800e070 con 0x7f3670138670 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366d59a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.968+0000 7f366d59a700 10 monclient: dump: 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:46.981 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 10 monclient: _finish_auth 0 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:46.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 1 -- 172.21.15.12:0/1746530918 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800be80 con 0x7f3670138670 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 5 monclient: authenticate success, global_id 4391 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 1 -- 172.21.15.12:0/1746530918 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36680092e0 con 0x7f3670138670 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366d59a700 10 monclient: dump: 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.983 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 4 set_mon_vals no callback set 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 10 monclient: shutdown 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 -- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 msgr2=0x7f367013aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3668009f50 tx=0x7f3668009840 comp rx=0 tx=0).stop 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 -- 172.21.15.12:0/1746530918 shutdown_connections 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3670138670 0x7f367013aa50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 --2- 172.21.15.12:0/1746530918 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3670133130 0x7f36700a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 -- 172.21.15.12:0/1746530918 >> 172.21.15.12:0/1746530918 conn(0x7f36700a6490 msgr2=0x7f36700a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 -- 172.21.15.12:0/1746530918 shutdown_connections 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.969+0000 7f366f59e700 1 -- 172.21.15.12:0/1746530918 wait complete. 2023-09-29T23:14:46.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 10 monclient: build_initial_monmap 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 10 monclient: monmap: 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:46.986 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 1 Processor -- start 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 1 -- start start 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 10 monclient: init 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.970+0000 7f366f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient: _add_conns ranks=[1,0,5,4,7,6,2,8,3] 2023-09-29T23:14:46.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 0x7f3670136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-f con 0x7f3670133130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f3670133500 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 0x7f367009f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): picked mon.noname-h con 0x7f3670138670 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670136440 con 0x7f3670133500 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670134140 con 0x7f3670133130 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f367009b3e0 con 0x7f3670138670 2023-09-29T23:14:46.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient(hunting): authenticate will time out at 1196.331078s 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54242/0 (socket says 172.21.15.12:54242) 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 -- 172.21.15.12:0/2664921020 learned_addr learned my addr 172.21.15.12:0/2664921020 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): get_auth_request con 0x7f3670133500 auth_method 0 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:46.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 0x7f3670136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 0x7f367009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 10 monclient(hunting): get_auth_request con 0x7f3670133130 auth_method 0 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 10 monclient(hunting): get_auth_request con 0x7f3670138670 auth_method 0 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2023-09-29T23:14:46.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 -- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 msgr2=0x7f367009f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 0x7f367009f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 -- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 msgr2=0x7f3670136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 0x7f3670136c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient: _finish_hunting 0 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 monclient: found mon.noname-a 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 -- 172.21.15.12:0/2664921020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660009420 con 0x7f3670133500 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366e59c700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 0x7f3670136c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366ed9d700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 0x7f367009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:46.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366dd9b700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f366400c590 tx=0x7f3664007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3664001120 con 0x7f3670133500 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 10 monclient: dump: 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:46.993 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 5 monclient: authenticate success, global_id 4563 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664011a10 con 0x7f3670133500 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36640102e0 con 0x7f3670133500 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient: _renew_subs 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3668009ae0 con 0x7f3670133500 2023-09-29T23:14:46.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f365e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f365e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.971+0000 7f366cd99700 4 set_mon_vals no callback set 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f365e7fc700 10 monclient: dump: 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:46.995 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 10 monclient: _renew_subs 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3670135e10 con 0x7f3670133500 2023-09-29T23:14:46.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3670070bc0 con 0x7f3670133500 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.972+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3664011b80 con 0x7f3670133500 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.973+0000 7f365e7fc700 1 --2- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 0x7f3650038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.973+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f36640463b0 con 0x7f3670133500 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.973+0000 7f366e59c700 1 --2- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 0x7f3650038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.973+0000 7f366e59c700 10 monclient: get_auth_request con 0x7f3650036850 auth_method 0 2023-09-29T23:14:46.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.973+0000 7f366e59c700 1 --2- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 0x7f3650038d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36600093f0 tx=0x7f3660008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.977+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f366400f670 con 0x7f3670133500 2023-09-29T23:14:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.977+0000 7f365e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:46.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:46.977+0000 7f365e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:47.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f366f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:47.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f366f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:47.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f366f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f367009b020 con 0x7f3670133500 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f365e7fc700 1 -- 172.21.15.12:0/2664921020 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3664017080 con 0x7f3670133500 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f365e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.218+0000 7f365e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:47.219 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 msgr2=0x7f3650038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 0x7f3650038d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36600093f0 tx=0x7f3660008040 comp rx=0 tx=0).stop 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 10 monclient: shutdown 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 msgr2=0x7f367009ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f366400c590 tx=0x7f3664007f10 comp rx=0 tx=0).stop 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 shutdown_connections 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> v2:172.21.15.153:6812/83257 conn(0x7f3650036850 0x7f3650038d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3670138670 0x7f367009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:47.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3670133130 0x7f3670136c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:47.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 --2- 172.21.15.12:0/2664921020 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3670133500 0x7f367009ecc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:47.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 >> 172.21.15.12:0/2664921020 conn(0x7f36700a6490 msgr2=0x7f367012dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:47.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 shutdown_connections 2023-09-29T23:14:47.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:47.220+0000 7f366f59e700 1 -- 172.21.15.12:0/2664921020 wait complete. 2023-09-29T23:14:50.233 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 10 monclient: get_monmap_and_config 2023-09-29T23:14:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 10 monclient: build_initial_monmap 2023-09-29T23:14:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 10 monclient: monmap: 2023-09-29T23:14:50.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:50.366010+0000 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:50.366010+0000 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:50.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:50.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 1 Processor -- start 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 1 -- start start 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.365+0000 7fe975d6b700 10 monclient: init 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient: _add_conns ranks=[7,10,4,5,8,3,6,9,0,1,2] 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe97009d5a0 0x7fe97009d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-i con 0x7fe97009d5a0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe97009cba0 0x7fe97009deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-f con 0x7fe97009cba0 addr v2:172.21.15.153:3300/0 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-c con 0x7fe97009adf0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97009bed0 con 0x7fe97009cba0 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe97009bbf0 con 0x7fe97009adf0 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe97009e3f0 con 0x7fe97009d5a0 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe975d6b700 10 monclient(hunting): authenticate will time out at 1199.726453s 2023-09-29T23:14:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:57466/0 (socket says 172.21.15.12:57466) 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 -- 172.21.15.12:0/1612234272 learned_addr learned my addr 172.21.15.12:0/1612234272 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): get_auth_request con 0x7fe97009adf0 auth_method 0 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:50.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96effd700 1 --2- 172.21.15.12:0/1612234272 >> v2:172.21.15.153:3300/0 conn(0x7fe97009cba0 0x7fe97009deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96f7fe700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe97009d5a0 0x7fe97009d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96effd700 10 monclient(hunting): get_auth_request con 0x7fe97009cba0 auth_method 0 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96f7fe700 10 monclient(hunting): get_auth_request con 0x7fe97009d5a0 auth_method 0 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient(hunting): handle_auth_done global_id 4495 payload 274 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 -- 172.21.15.12:0/1612234272 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe97009d5a0 msgr2=0x7fe97009d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe97009d5a0 0x7fe97009d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 -- 172.21.15.12:0/1612234272 >> v2:172.21.15.153:3300/0 conn(0x7fe97009cba0 msgr2=0x7fe97009deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 --2- 172.21.15.12:0/1612234272 >> v2:172.21.15.153:3300/0 conn(0x7fe97009cba0 0x7fe97009deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 monclient: found mon.noname-c 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.366+0000 7fe96ffff700 1 -- 172.21.15.12:0/1612234272 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe970068ed0 con 0x7fe97009adf0 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96ffff700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f20 tx=0x7fe968009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=460e7eaaab032b3e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 1 -- 172.21.15.12:0/1612234272 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe96800e070 con 0x7fe97009adf0 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:14:50.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: dump: 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:50.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:50.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 5 monclient: authenticate success, global_id 4495 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 1 -- 172.21.15.12:0/1612234272 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968007400 con 0x7fe97009adf0 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 1 -- 172.21.15.12:0/1612234272 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe9680092e0 con 0x7fe97009adf0 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 4 set_mon_vals no callback set 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:50.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe96e7fc700 10 monclient: dump: 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 10 monclient: shutdown 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 -- 172.21.15.12:0/1612234272 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 msgr2=0x7fe9701333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f20 tx=0x7fe968009840 comp rx=0 tx=0).stop 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 -- 172.21.15.12:0/1612234272 shutdown_connections 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe97009adf0 0x7fe9701333b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 --2- 172.21.15.12:0/1612234272 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe97009d5a0 0x7fe97009d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 --2- 172.21.15.12:0/1612234272 >> v2:172.21.15.153:3300/0 conn(0x7fe97009cba0 0x7fe97009deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 -- 172.21.15.12:0/1612234272 >> 172.21.15.12:0/1612234272 conn(0x7fe97006a370 msgr2=0x7fe9700a2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.367+0000 7fe975d6b700 1 -- 172.21.15.12:0/1612234272 shutdown_connections 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.368+0000 7fe975d6b700 1 -- 172.21.15.12:0/1612234272 wait complete. 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.368+0000 7fe975d6b700 10 monclient: build_initial_monmap 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.368+0000 7fe975d6b700 10 monclient: monmap: 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 Processor -- start 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 -- start start 2023-09-29T23:14:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient: init 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient: _add_conns ranks=[1,5,8,7,0,3,2,4,6] 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 0x7fe97013eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-f con 0x7fe97009adf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe97009d5a0 0x7fe97013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-h con 0x7fe97009d5a0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): picked mon.noname-e con 0x7fe970120d30 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970123590 con 0x7fe97009adf0 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe970121880 con 0x7fe97009d5a0 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fe97013f6c0 con 0x7fe970120d30 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe975d6b700 10 monclient(hunting): authenticate will time out at 1199.729623s 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56410/0 (socket says 172.21.15.12:56410) 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 1 -- 172.21.15.12:0/460323880 learned_addr learned my addr 172.21.15.12:0/460323880 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): get_auth_request con 0x7fe970120d30 auth_method 0 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96f7fe700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 0x7fe97013eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96effd700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe97009d5a0 0x7fe97013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96f7fe700 10 monclient(hunting): get_auth_request con 0x7fe97009adf0 auth_method 0 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.369+0000 7fe96effd700 10 monclient(hunting): get_auth_request con 0x7fe97009d5a0 auth_method 0 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 -- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe97009d5a0 msgr2=0x7fe97013e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe97009d5a0 0x7fe97013e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 -- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 msgr2=0x7fe97013eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 0x7fe97013eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 monclient: found mon.noname-e 2023-09-29T23:14:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 -- 172.21.15.12:0/460323880 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968009ae0 con 0x7fe970120d30 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96f7fe700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 0x7fe97013eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96ffff700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe968009880 tx=0x7fe968018000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe968026460 con 0x7fe970120d30 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: dump: 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 5 monclient: authenticate success, global_id 4400 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968026600 con 0x7fe970120d30 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 10 monclient: _renew_subs 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe968026bf0 con 0x7fe970120d30 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97004c960 con 0x7fe970120d30 2023-09-29T23:14:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe96dffb700 4 set_mon_vals no callback set 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe9577fe700 10 monclient: dump: 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 10 monclient: _renew_subs 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.370+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe970123870 con 0x7fe970120d30 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe968003040 con 0x7fe970120d30 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe975d6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe975d6b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe975d6b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970070bc0 con 0x7fe970120d30 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe9577fe700 1 --2- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 0x7fe95c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe968049070 con 0x7fe970120d30 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe96f7fe700 1 --2- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 0x7fe95c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.371+0000 7fe96f7fe700 10 monclient: get_auth_request con 0x7fe95c036880 auth_method 0 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.372+0000 7fe96f7fe700 1 --2- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 0x7fe95c038d30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe960006fd0 tx=0x7fe960008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.376+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe96801c9a0 con 0x7fe970120d30 2023-09-29T23:14:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.376+0000 7fe9577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.376+0000 7fe9577fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe975d6b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe975d6b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe975d6b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe97009edc0 con 0x7fe970120d30 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe9577fe700 1 -- 172.21.15.12:0/460323880 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe968021020 con 0x7fe970120d30 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe9577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.612+0000 7fe9577fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:50.613 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 msgr2=0x7fe95c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 0x7fe95c038d30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe960006fd0 tx=0x7fe960008040 comp rx=0 tx=0).stop 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 10 monclient: shutdown 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 msgr2=0x7fe9701420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe968009880 tx=0x7fe968018000 comp rx=0 tx=0).stop 2023-09-29T23:14:50.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 shutdown_connections 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> v2:172.21.15.153:6812/83257 conn(0x7fe95c036880 0x7fe95c038d30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe970120d30 0x7fe9701420d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97009adf0 0x7fe97013eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 --2- 172.21.15.12:0/460323880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe97009d5a0 0x7fe97013e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 >> 172.21.15.12:0/460323880 conn(0x7fe97006a370 msgr2=0x7fe9700a1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 shutdown_connections 2023-09-29T23:14:50.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:50.615+0000 7fe975d6b700 1 -- 172.21.15.12:0/460323880 wait complete. 2023-09-29T23:14:53.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.756+0000 7f0354564700 10 monclient: get_monmap_and_config 2023-09-29T23:14:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.756+0000 7f0354564700 10 monclient: build_initial_monmap 2023-09-29T23:14:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.756+0000 7f0354564700 10 monclient: monmap: 2023-09-29T23:14:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:53.757 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:53.757702+0000 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:53.757702+0000 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:53.758 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.757+0000 7f0354564700 1 Processor -- start 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.757+0000 7f0354564700 1 -- start start 2023-09-29T23:14:53.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.757+0000 7f0354564700 10 monclient: init 2023-09-29T23:14:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient: _add_conns ranks=[7,9,2,0,6,1,10,3,4,8,5] 2023-09-29T23:14:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f034c0a1f90 0x7f034c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): picked mon.noname-i con 0x7f034c0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:53.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): picked mon.noname-a con 0x7f034c0a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): picked mon.noname-b con 0x7f034c133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c09e350 con 0x7f034c0a2360 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c09ef70 con 0x7f034c133740 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f034c09da70 con 0x7f034c0a1f90 2023-09-29T23:14:53.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0354564700 10 monclient(hunting): authenticate will time out at 1203.118123s 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41688/0 (socket says 172.21.15.12:41688) 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 -- 172.21.15.12:0/4203152502 learned_addr learned my addr 172.21.15.12:0/4203152502 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): get_auth_request con 0x7f034c133740 auth_method 0 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51846/0 (socket says 172.21.15.12:51846) 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): get_auth_request con 0x7f034c0a2360 auth_method 0 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352300700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f034c0a1f90 0x7f034c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352300700 10 monclient(hunting): get_auth_request con 0x7f034c0a1f90 auth_method 0 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352300700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352300700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352300700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 -- 172.21.15.12:0/4203152502 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f034c0a1f90 msgr2=0x7f034c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f034c0a1f90 0x7f034c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 -- 172.21.15.12:0/4203152502 >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 msgr2=0x7f034c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 --2- 172.21.15.12:0/4203152502 >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient: _finish_hunting 0 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 monclient: found mon.noname-b 2023-09-29T23:14:53.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 -- 172.21.15.12:0/4203152502 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c068ed0 con 0x7f034c133740 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0351aff700 1 --2- 172.21.15.12:0/4203152502 >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f0352b01700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0344009f50 tx=0x7f0344009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347212e588b2b5 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f03512fe700 1 -- 172.21.15.12:0/4203152502 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f034400e070 con 0x7f034c133740 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f03512fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f03512fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.758+0000 7f03512fe700 10 monclient: dump: 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:53.773 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:53.774 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 10 monclient: _finish_auth 0 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 1 -- 172.21.15.12:0/4203152502 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034400be40 con 0x7f034c133740 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 5 monclient: authenticate success, global_id 4412 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 1 -- 172.21.15.12:0/4203152502 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f03440092e0 con 0x7f034c133740 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f03512fe700 10 monclient: dump: 2023-09-29T23:14:53.775 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:53.776 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 4 set_mon_vals no callback set 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 10 monclient: shutdown 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 -- 172.21.15.12:0/4203152502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 msgr2=0x7f034c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0344009f50 tx=0x7f0344009840 comp rx=0 tx=0).stop 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 -- 172.21.15.12:0/4203152502 shutdown_connections 2023-09-29T23:14:53.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c133be0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 --2- 172.21.15.12:0/4203152502 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f034c0a1f90 0x7f034c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 --2- 172.21.15.12:0/4203152502 >> v2:172.21.15.12:3300/0 conn(0x7f034c0a2360 0x7f034c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 -- 172.21.15.12:0/4203152502 >> 172.21.15.12:0/4203152502 conn(0x7f034c0a3b50 msgr2=0x7f034c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 -- 172.21.15.12:0/4203152502 shutdown_connections 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.759+0000 7f0354564700 1 -- 172.21.15.12:0/4203152502 wait complete. 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.760+0000 7f0354564700 10 monclient: build_initial_monmap 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.760+0000 7f0354564700 10 monclient: monmap: 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:53.778 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.760+0000 7f0354564700 1 Processor -- start 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.760+0000 7f0354564700 1 -- start start 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.760+0000 7f0354564700 10 monclient: init 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient: _add_conns ranks=[3,1,2,7,5,4,6,0,8] 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c0a1f90 0x7f034c09c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): picked mon.noname-g con 0x7f034c0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f034c0a2360 0x7f034c09c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): picked mon.noname-f con 0x7f034c0a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): picked mon.noname-b con 0x7f034c133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c09ffc0 con 0x7f034c0a2360 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c130f10 con 0x7f034c133740 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c12eec0 con 0x7f034c0a1f90 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient(hunting): authenticate will time out at 1203.121120s 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41698/0 (socket says 172.21.15.12:41698) 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 -- 172.21.15.12:0/2618497220 learned_addr learned my addr 172.21.15.12:0/2618497220 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): get_auth_request con 0x7f034c133740 auth_method 0 2023-09-29T23:14:53.781 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0351aff700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f034c0a2360 0x7f034c09c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352300700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c0a1f90 0x7f034c09c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352300700 10 monclient(hunting): get_auth_request con 0x7f034c0a1f90 auth_method 0 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352300700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352300700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352300700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0351aff700 10 monclient(hunting): get_auth_request con 0x7f034c0a2360 auth_method 0 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0351aff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0351aff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0351aff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 -- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c0a1f90 msgr2=0x7f034c09c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c0a1f90 0x7f034c09c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 -- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f034c0a2360 msgr2=0x7f034c09c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f034c0a2360 0x7f034c09c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient: _finish_hunting 0 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 monclient: found mon.noname-b 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:53.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 -- 172.21.15.12:0/2618497220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c009420 con 0x7f034c133740 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0352b01700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0344007730 tx=0x7f0344018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f03440092e0 con 0x7f034c133740 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: dump: 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:53.784 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:53.785 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: _finish_auth 0 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034400bc90 con 0x7f034c133740 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 5 monclient: authenticate success, global_id 4421 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient: _renew_subs 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0344009ae0 con 0x7f034c133740 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0344007000 con 0x7f034c133740 2023-09-29T23:14:53.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0350afd700 4 set_mon_vals no callback set 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f03427fc700 10 monclient: dump: 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:53.787 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.761+0000 7f0354564700 10 monclient: _renew_subs 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:53.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c04c960 con 0x7f034c133740 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0344023070 con 0x7f034c133740 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f03427fc700 1 --2- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 0x7f0330038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.762+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c070bc0 con 0x7f034c133740 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.763+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f03440491c0 con 0x7f034c133740 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.763+0000 7f0352300700 1 --2- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 0x7f0330038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.763+0000 7f0352300700 10 monclient: get_auth_request con 0x7f0330036880 auth_method 0 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.763+0000 7f0352300700 1 --2- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 0x7f0330038d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0348006fd0 tx=0x7f0348008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.767+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0344007e60 con 0x7f034c133740 2023-09-29T23:14:53.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.767+0000 7f03427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:53.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:53.767+0000 7f03427fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:54.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f0354564700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:54.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f0354564700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:54.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f0354564700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:54.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f034c131050 con 0x7f034c133740 2023-09-29T23:14:54.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f03427fc700 1 -- 172.21.15.12:0/2618497220 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f034c131050 con 0x7f034c133740 2023-09-29T23:14:54.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f03427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:54.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.004+0000 7f03427fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:54.005 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:54.005 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 msgr2=0x7f0330038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 0x7f0330038d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0348006fd0 tx=0x7f0348008040 comp rx=0 tx=0).stop 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 10 monclient: shutdown 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 msgr2=0x7f034c09cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0344007730 tx=0x7f0344018000 comp rx=0 tx=0).stop 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 shutdown_connections 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> v2:172.21.15.153:6812/83257 conn(0x7f0330036880 0x7f0330038d30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:54.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f034c133740 0x7f034c09cf00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c0a1f90 0x7f034c09c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 --2- 172.21.15.12:0/2618497220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f034c0a2360 0x7f034c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 >> 172.21.15.12:0/2618497220 conn(0x7f034c0a3b50 msgr2=0x7f034c06f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 shutdown_connections 2023-09-29T23:14:54.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:54.006+0000 7f0354564700 1 -- 172.21.15.12:0/2618497220 wait complete. 2023-09-29T23:14:57.019 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 10 monclient: get_monmap_and_config 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 10 monclient: build_initial_monmap 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 10 monclient: monmap: 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:57.155 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:14:57.155983+0000 2023-09-29T23:14:57.156 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:14:57.155983+0000 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:57.157 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:14:57.158 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 1 Processor -- start 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 1 -- start start 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.155+0000 7ff67e59c700 10 monclient: init 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient: _add_conns ranks=[7,8,2,1,4,10,9,6,3,5,0] 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 0x7ff67809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-i con 0x7ff6780a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-e con 0x7ff6780a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:14:57.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-b con 0x7ff678133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67809e350 con 0x7ff678133740 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff67809ef70 con 0x7ff6780a1f90 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff67809da70 con 0x7ff6780a2360 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67e59c700 10 monclient(hunting): authenticate will time out at 1206.516400s 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:34396/0 (socket says 172.21.15.12:34396) 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41708/0 (socket says 172.21.15.12:41708) 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 -- 172.21.15.12:0/3196985328 learned_addr learned my addr 172.21.15.12:0/3196985328 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): get_auth_request con 0x7ff6780a2360 auth_method 0 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): get_auth_request con 0x7ff678133740 auth_method 0 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:57.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff677fff700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 0x7ff67809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 -- 172.21.15.12:0/3196985328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 msgr2=0x7ff67809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 0x7ff67809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 -- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 msgr2=0x7ff678133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient: _finish_hunting 0 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 monclient: found mon.noname-e 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff6777fe700 1 -- 172.21.15.12:0/3196985328 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678068ed0 con 0x7ff6780a2360 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff677fff700 10 monclient: get_auth_request con 0x7ff6780a1f90 auth_method 0 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff67cb39700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.156+0000 7ff677fff700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 0x7ff67809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff6777fe700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff660009a40 tx=0x7ff660009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=16e90b6a60352570 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 1 -- 172.21.15.12:0/3196985328 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff66000bc60 con 0x7ff6780a2360 2023-09-29T23:14:57.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: dump: 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:57.172 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: _finish_auth 0 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 1 -- 172.21.15.12:0/3196985328 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66000bde0 con 0x7ff6780a2360 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 5 monclient: authenticate success, global_id 4409 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:14:57.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 10 monclient: get_monmap_and_config success 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 1 -- 172.21.15.12:0/3196985328 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff6600118a0 con 0x7ff6780a2360 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 4 set_mon_vals no callback set 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff676ffd700 10 monclient: dump: 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.174 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 10 monclient: shutdown 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 -- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 msgr2=0x7ff6780a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff660009a40 tx=0x7ff660009250 comp rx=0 tx=0).stop 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 -- 172.21.15.12:0/3196985328 shutdown_connections 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff678133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff6780a1f90 0x7ff67809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3196985328 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff6780a2360 0x7ff6780a27b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 -- 172.21.15.12:0/3196985328 >> 172.21.15.12:0/3196985328 conn(0x7ff6780a3b50 msgr2=0x7ff6780a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 -- 172.21.15.12:0/3196985328 shutdown_connections 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.157+0000 7ff67e59c700 1 -- 172.21.15.12:0/3196985328 wait complete. 2023-09-29T23:14:57.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.158+0000 7ff67e59c700 10 monclient: build_initial_monmap 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.158+0000 7ff67e59c700 10 monclient: monmap: 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:14:57.177 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.158+0000 7ff67e59c700 1 Processor -- start 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.158+0000 7ff67e59c700 1 -- start start 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.158+0000 7ff67e59c700 10 monclient: init 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient: _reopen_session rank -1 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient: _add_conns ranks=[3,4,2,5,8,6,0,1,7] 2023-09-29T23:14:57.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 0x7ff67809c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-g con 0x7ff6780a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-c con 0x7ff6780a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): picked mon.noname-b con 0x7ff678133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67809e510 con 0x7ff678133740 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff678130d90 con 0x7ff6780a1f90 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff678130ab0 con 0x7ff6780a2360 2023-09-29T23:14:57.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): _renew_subs 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67e59c700 10 monclient(hunting): authenticate will time out at 1206.519450s 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41716/0 (socket says 172.21.15.12:41716) 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:38446/0 (socket says 172.21.15.12:38446) 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 -- 172.21.15.12:0/3755884738 learned_addr learned my addr 172.21.15.12:0/3755884738 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): get_auth_request con 0x7ff678133740 auth_method 0 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): get_auth_request con 0x7ff6780a2360 auth_method 0 2023-09-29T23:14:57.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 0x7ff67809c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 10 monclient(hunting): get_auth_request con 0x7ff6780a1f90 auth_method 0 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:14:57.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 -- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 msgr2=0x7ff67809cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 -- 172.21.15.12:0/3755884738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 msgr2=0x7ff67809c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 0x7ff67809c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient: _finish_hunting 0 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 monclient: found mon.noname-b 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff67cb39700 1 -- 172.21.15.12:0/3755884738 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670009a20 con 0x7ff678133740 2023-09-29T23:14:57.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff677fff700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 0x7ff67809c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.159+0000 7ff6777fe700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67cb39700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ef0 tx=0x7ff67000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff67000cc20 con 0x7ff678133740 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: dump: 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.183 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:57.184 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: _finish_auth 0 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000f040 con 0x7ff678133740 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 5 monclient: authenticate success, global_id 4439 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 10 monclient: _renew_subs 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6600094f0 con 0x7ff678133740 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff67000cdc0 con 0x7ff678133740 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:14:57.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff66ffff700 10 monclient: dump: 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:14:57.186 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff6767fc700 4 set_mon_vals no callback set 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 10 monclient: _renew_subs 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.160+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67804c960 con 0x7ff678133740 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff670014480 con 0x7ff678133740 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff67e59c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:57.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff67e59c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff67e59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff678070bc0 con 0x7ff678133740 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff66ffff700 1 --2- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 0x7ff658038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff670047f60 con 0x7ff678133740 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff677fff700 1 --2- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 0x7ff658038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.161+0000 7ff677fff700 10 monclient: get_auth_request con 0x7ff658036850 auth_method 0 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.162+0000 7ff677fff700 1 --2- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 0x7ff658038d00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff66800ac10 tx=0x7ff668006f90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.166+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6700795a0 con 0x7ff678133740 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.166+0000 7ff66ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:14:57.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.166+0000 7ff66ffff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.402+0000 7ff67e59c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.402+0000 7ff67e59c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.402+0000 7ff67e59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.402+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff678066790 con 0x7ff678133740 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.403+0000 7ff66ffff700 1 -- 172.21.15.12:0/3755884738 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff670018070 con 0x7ff678133740 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.403+0000 7ff66ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.403+0000 7ff66ffff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:14:57.403 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:14:57.404 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:14:57.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 msgr2=0x7ff658038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:57.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 0x7ff658038d00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff66800ac10 tx=0x7ff668006f90 comp rx=0 tx=0).stop 2023-09-29T23:14:57.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 10 monclient: shutdown 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 msgr2=0x7ff67812d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ef0 tx=0x7ff67000b040 comp rx=0 tx=0).stop 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 shutdown_connections 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> v2:172.21.15.153:6812/83257 conn(0x7ff658036850 0x7ff658038d00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff678133740 0x7ff67812d350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6780a1f90 0x7ff67809c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 --2- 172.21.15.12:0/3755884738 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff6780a2360 0x7ff67809cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 >> 172.21.15.12:0/3755884738 conn(0x7ff6780a3b50 msgr2=0x7ff67806f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:14:57.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 shutdown_connections 2023-09-29T23:14:57.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:14:57.405+0000 7ff67e59c700 1 -- 172.21.15.12:0/3755884738 wait complete. 2023-09-29T23:15:00.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 10 monclient: get_monmap_and_config 2023-09-29T23:15:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 10 monclient: build_initial_monmap 2023-09-29T23:15:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 10 monclient: monmap: 2023-09-29T23:15:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:00.549157+0000 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:00.549157+0000 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:00.550 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 1 Processor -- start 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 1 -- start start 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.548+0000 7fcc49153700 10 monclient: init 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient: _add_conns ranks=[6,8,10,1,9,7,5,0,3,4,2] 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-d con 0x7fcc4406ef50 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:00.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 0x7fcc4406f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-e con 0x7fcc4412d000 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc441300b0 0x7fcc44130510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-f con 0x7fcc441300b0 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4406d280 con 0x7fcc441300b0 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4406d460 con 0x7fcc4406ef50 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4406da20 con 0x7fcc4412d000 2023-09-29T23:15:00.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc49153700 10 monclient(hunting): authenticate will time out at 1209.909660s 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 0x7fcc4406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:60696/0 (socket says 172.21.15.12:60696) 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 1 -- 172.21.15.12:0/2135649785 learned_addr learned my addr 172.21.15.12:0/2135649785 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): get_auth_request con 0x7fcc4406ef50 auth_method 0 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): get_auth_request con 0x7fcc4412d000 auth_method 0 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:00.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.549+0000 7fcc4359e700 1 --2- 172.21.15.12:0/2135649785 >> v2:172.21.15.153:3300/0 conn(0x7fcc441300b0 0x7fcc44130510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc4359e700 10 monclient(hunting): get_auth_request con 0x7fcc441300b0 auth_method 0 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc4359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc4359e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc4359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 -- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 msgr2=0x7fcc4406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 0x7fcc4406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 -- 172.21.15.12:0/2135649785 >> v2:172.21.15.153:3300/0 conn(0x7fcc441300b0 msgr2=0x7fcc44130510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:00.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 --2- 172.21.15.12:0/2135649785 >> v2:172.21.15.153:3300/0 conn(0x7fcc441300b0 0x7fcc44130510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 monclient: found mon.noname-d 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 -- 172.21.15.12:0/2135649785 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44068ed0 con 0x7fcc4406ef50 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc4259c700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 0x7fcc4406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc42d9d700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc340099c0 tx=0x7fcc34009220 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1e03add18f031525 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 1 -- 172.21.15.12:0/2135649785 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc34005c10 con 0x7fcc4406ef50 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:00.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: dump: 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:00.567 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: _finish_auth 0 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 1 -- 172.21.15.12:0/2135649785 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc34005d90 con 0x7fcc4406ef50 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 5 monclient: authenticate success, global_id 4533 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:00.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 1 -- 172.21.15.12:0/2135649785 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc34006030 con 0x7fcc4406ef50 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 4 set_mon_vals no callback set 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc41d9b700 10 monclient: dump: 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:00.569 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 10 monclient: shutdown 2023-09-29T23:15:00.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 -- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 msgr2=0x7fcc4412d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc340099c0 tx=0x7fcc34009220 comp rx=0 tx=0).stop 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 -- 172.21.15.12:0/2135649785 shutdown_connections 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 --2- 172.21.15.12:0/2135649785 >> v2:172.21.15.153:3300/0 conn(0x7fcc441300b0 0x7fcc44130510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fcc4406ef50 0x7fcc4412d630 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 --2- 172.21.15.12:0/2135649785 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc4412d000 0x7fcc4406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 -- 172.21.15.12:0/2135649785 >> 172.21.15.12:0/2135649785 conn(0x7fcc440a3330 msgr2=0x7fcc440a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 -- 172.21.15.12:0/2135649785 shutdown_connections 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.550+0000 7fcc49153700 1 -- 172.21.15.12:0/2135649785 wait complete. 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.551+0000 7fcc49153700 10 monclient: build_initial_monmap 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.551+0000 7fcc49153700 10 monclient: monmap: 2023-09-29T23:15:00.571 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:00.572 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.551+0000 7fcc49153700 1 Processor -- start 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 -- start start 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient: init 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient: _add_conns ranks=[4,2,8,3,7,6,0,5,1] 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 0x7fcc4411ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-c con 0x7fcc4406ef50 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc4412d000 0x7fcc4413f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-b con 0x7fcc4412d000 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): picked mon.noname-e con 0x7fcc441300b0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4411ba00 con 0x7fcc4412d000 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4409fa40 con 0x7fcc4406ef50 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4409f790 con 0x7fcc441300b0 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc49153700 10 monclient(hunting): authenticate will time out at 1209.912485s 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 0x7fcc4411ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:34418/0 (socket says 172.21.15.12:34418) 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 0x7fcc4411ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:38452/0 (socket says 172.21.15.12:38452) 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 -- 172.21.15.12:0/766751548 learned_addr learned my addr 172.21.15.12:0/766751548 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): get_auth_request con 0x7fcc441300b0 auth_method 0 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): get_auth_request con 0x7fcc4406ef50 auth_method 0 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc4412d000 0x7fcc4413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): get_auth_request con 0x7fcc4412d000 auth_method 0 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc42d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:00.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 -- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 msgr2=0x7fcc4411ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 0x7fcc4411ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 -- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc4412d000 msgr2=0x7fcc4413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc4412d000 0x7fcc4413f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient: _finish_hunting 0 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 monclient: found mon.noname-e 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:15:00.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.552+0000 7fcc4359e700 1 -- 172.21.15.12:0/766751548 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38009420 con 0x7fcc441300b0 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc4359e700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c007f20 tx=0x7fcc3c009860 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc3c0098f0 con 0x7fcc441300b0 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: dump: 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:00.578 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: _finish_auth 0 2023-09-29T23:15:00.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c00eb90 con 0x7fcc441300b0 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 5 monclient: authenticate success, global_id 4427 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 10 monclient: _renew_subs 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc34009480 con 0x7fcc441300b0 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcc3c014d20 con 0x7fcc441300b0 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc2affd700 10 monclient: dump: 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:00.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:00.581 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc4159a700 4 set_mon_vals no callback set 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 10 monclient: _renew_subs 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.553+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4404c960 con 0x7fcc441300b0 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fcc3c00ed00 con 0x7fcc441300b0 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc49153700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:00.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc49153700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc49153700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4411bf20 con 0x7fcc441300b0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc2affd700 1 --2- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 0x7fcc30038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcc3c047860 con 0x7fcc441300b0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc42d9d700 1 --2- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 0x7fcc30038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.554+0000 7fcc42d9d700 10 monclient: get_auth_request con 0x7fcc30036850 auth_method 0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.555+0000 7fcc42d9d700 1 --2- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 0x7fcc30038d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc34007fd0 tx=0x7fcc34016000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.559+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc4411bf20 con 0x7fcc441300b0 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.559+0000 7fcc2affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:00.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.559+0000 7fcc2affd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:00.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.792+0000 7fcc49153700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:00.793 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.792+0000 7fcc49153700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:00.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.792+0000 7fcc49153700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:00.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.792+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc4411bf20 con 0x7fcc441300b0 2023-09-29T23:15:00.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.793+0000 7fcc2affd700 1 -- 172.21.15.12:0/766751548 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc4411bf20 con 0x7fcc441300b0 2023-09-29T23:15:00.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.793+0000 7fcc2affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:00.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.793+0000 7fcc2affd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:00.795 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:00.795 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:00.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 msgr2=0x7fcc30038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:00.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 0x7fcc30038d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc34007fd0 tx=0x7fcc34016000 comp rx=0 tx=0).stop 2023-09-29T23:15:00.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 10 monclient: shutdown 2023-09-29T23:15:00.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:00.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 msgr2=0x7fcc4413fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:00.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c007f20 tx=0x7fcc3c009860 comp rx=0 tx=0).stop 2023-09-29T23:15:00.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 shutdown_connections 2023-09-29T23:15:00.798 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fcc441300b0 0x7fcc4413fa50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.798 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcc4406ef50 0x7fcc4411ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.798 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> v2:172.21.15.153:6812/83257 conn(0x7fcc30036850 0x7fcc30038d00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.798 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 --2- 172.21.15.12:0/766751548 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcc4412d000 0x7fcc4413f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:00.798 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.796+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 >> 172.21.15.12:0/766751548 conn(0x7fcc440a3330 msgr2=0x7fcc4412ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:00.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.797+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 shutdown_connections 2023-09-29T23:15:00.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:00.797+0000 7fcc49153700 1 -- 172.21.15.12:0/766751548 wait complete. 2023-09-29T23:15:03.809 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:03.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.935+0000 7f32929d0700 10 monclient: get_monmap_and_config 2023-09-29T23:15:03.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.935+0000 7f32929d0700 10 monclient: build_initial_monmap 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.935+0000 7f32929d0700 10 monclient: monmap: 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:03.936428+0000 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:03.936428+0000 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:03.936 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:03.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 Processor -- start 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 -- start start 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient: init 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient: _add_conns ranks=[7,4,6,10,9,5,0,1,8,2,3] 2023-09-29T23:15:03.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-i con 0x7f328c0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-c con 0x7f328c0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-d con 0x7f328c133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09e350 con 0x7f328c0a2360 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09ef70 con 0x7f328c133740 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09da70 con 0x7f328c0a1f90 2023-09-29T23:15:03.947 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f32929d0700 10 monclient(hunting): authenticate will time out at 1213.296818s 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f3290f6d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.936+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:50556/0 (socket says 172.21.15.12:50556) 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37236/0 (socket says 172.21.15.12:37236) 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 -- 172.21.15.12:0/3542662877 learned_addr learned my addr 172.21.15.12:0/3542662877 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): get_auth_request con 0x7f328c133740 auth_method 0 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): get_auth_request con 0x7f328c0a2360 auth_method 0 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328bfff700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328bfff700 10 monclient(hunting): get_auth_request con 0x7f328c0a1f90 auth_method 0 2023-09-29T23:15:03.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient(hunting): handle_auth_done global_id 4542 payload 274 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 -- 172.21.15.12:0/3542662877 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f328c0a1f90 msgr2=0x7f328c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 -- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 msgr2=0x7f328c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient: _finish_hunting 0 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 monclient: found mon.noname-d 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:03.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 -- 172.21.15.12:0/3542662877 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c068ed0 con 0x7f328c133740 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328b7fe700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f3290f6d700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3284009f80 tx=0x7f3284009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c3d45e0447829284 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 1 -- 172.21.15.12:0/3542662877 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f328400e070 con 0x7f328c133740 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: dump: 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.951 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:03.952 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: _finish_auth 0 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f32929d0700 5 monclient: authenticate success, global_id 4542 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f32929d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 1 -- 172.21.15.12:0/3542662877 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284007400 con 0x7f328c133740 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f32929d0700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f32929d0700 4 set_mon_vals no callback set 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 1 -- 172.21.15.12:0/3542662877 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f32840092e0 con 0x7f328c133740 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:03.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.937+0000 7f328affd700 10 monclient: dump: 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:03.954 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 10 monclient: shutdown 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 -- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 msgr2=0x7f328c133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3284009f80 tx=0x7f3284009840 comp rx=0 tx=0).stop 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 -- 172.21.15.12:0/3542662877 shutdown_connections 2023-09-29T23:15:03.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f328c133740 0x7f328c133bb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f328c0a1f90 0x7f328c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 --2- 172.21.15.12:0/3542662877 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f328c0a2360 0x7f328c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 -- 172.21.15.12:0/3542662877 >> 172.21.15.12:0/3542662877 conn(0x7f328c0a3b50 msgr2=0x7f328c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 -- 172.21.15.12:0/3542662877 shutdown_connections 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 1 -- 172.21.15.12:0/3542662877 wait complete. 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 10 monclient: build_initial_monmap 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.938+0000 7f32929d0700 10 monclient: monmap: 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:03.956 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:03.957 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 Processor -- start 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 -- start start 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient: init 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient: _add_conns ranks=[1,3,5,0,7,4,6,2,8] 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 0x7f328c142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-f con 0x7f328c0a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-g con 0x7f328c0a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): picked mon.noname-h con 0x7f328c133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c143330 con 0x7f328c0a1f90 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09ffc0 con 0x7f328c0a2360 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f328c1312e0 con 0x7f328c133740 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.939+0000 7f32929d0700 10 monclient(hunting): authenticate will time out at 1213.299870s 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 0x7f328c142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:59672/0 (socket says 172.21.15.12:59672) 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.12:39170/0 (socket says 172.21.15.12:39170) 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 1 -- 172.21.15.12:0/2727064872 learned_addr learned my addr 172.21.15.12:0/2727064872 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): get_auth_request con 0x7f328c133740 auth_method 0 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): get_auth_request con 0x7f328c0a2360 auth_method 0 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): get_auth_request con 0x7f328c0a1f90 auth_method 0 2023-09-29T23:15:03.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:03.961 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient(hunting): handle_auth_done global_id 4564 payload 824 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 -- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 msgr2=0x7f328c12d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 -- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 msgr2=0x7f328c132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 monclient: found mon.noname-f 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 -- 172.21.15.12:0/2727064872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c009420 con 0x7f328c0a1f90 2023-09-29T23:15:03.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328b7fe700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f3290f6d700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.940+0000 7f328bfff700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 0x7f328c142410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f328000d890 tx=0x7f3280008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f32800073c0 con 0x7f328c0a1f90 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: dump: 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:03.963 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: _finish_auth 0 2023-09-29T23:15:03.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3280012a70 con 0x7f328c0a1f90 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 5 monclient: authenticate success, global_id 4564 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 10 monclient: _renew_subs 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3284009ae0 con 0x7f328c0a1f90 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3280012be0 con 0x7f328c0a1f90 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f3273fff700 10 monclient: dump: 2023-09-29T23:15:03.965 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:03.966 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f328a7fc700 4 set_mon_vals no callback set 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 10 monclient: _renew_subs 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.941+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c04c960 con 0x7f328c0a1f90 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3280012d40 con 0x7f328c0a1f90 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f3273fff700 1 --2- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 0x7f3278038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3280047550 con 0x7f328c0a1f90 2023-09-29T23:15:03.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f32929d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f32929d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f32929d0700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c070bc0 con 0x7f328c0a1f90 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f328b7fe700 1 --2- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 0x7f3278038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f328b7fe700 10 monclient: get_auth_request con 0x7f3278036850 auth_method 0 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.942+0000 7f328b7fe700 1 --2- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 0x7f3278038d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f328c12ec80 tx=0x7f327c0091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:03.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.947+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f328001bd30 con 0x7f328c0a1f90 2023-09-29T23:15:03.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.947+0000 7f3273fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:03.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:03.947+0000 7f3273fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:04.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.183+0000 7f32929d0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:04.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.183+0000 7f32929d0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:04.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.183+0000 7f32929d0700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:15:04.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.184+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f328c0a5c90 con 0x7f328c0a1f90 2023-09-29T23:15:04.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.184+0000 7f3273fff700 1 -- 172.21.15.12:0/2727064872 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f328002b100 con 0x7f328c0a1f90 2023-09-29T23:15:04.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.184+0000 7f3273fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:04.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.184+0000 7f3273fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:04.185 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:04.185 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:04.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 msgr2=0x7f3278038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 0x7f3278038d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f328c12ec80 tx=0x7f327c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 10 monclient: shutdown 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 msgr2=0x7f328c142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 0x7f328c142410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f328000d890 tx=0x7f3280008040 comp rx=0 tx=0).stop 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 shutdown_connections 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f328c133740 0x7f328c12d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:04.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f328c0a1f90 0x7f328c142410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:04.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> v2:172.21.15.153:6812/83257 conn(0x7f3278036850 0x7f3278038d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:04.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 --2- 172.21.15.12:0/2727064872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f328c0a2360 0x7f328c132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:04.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 >> 172.21.15.12:0/2727064872 conn(0x7f328c0a3b50 msgr2=0x7f328c135dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:04.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 shutdown_connections 2023-09-29T23:15:04.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:04.186+0000 7f32929d0700 1 -- 172.21.15.12:0/2727064872 wait complete. 2023-09-29T23:15:07.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:07.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.326+0000 7f0805803700 10 monclient: get_monmap_and_config 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.326+0000 7f0805803700 10 monclient: build_initial_monmap 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.326+0000 7f0805803700 10 monclient: monmap: 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:07.327779+0000 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:07.327779+0000 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:07.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:07.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 1 Processor -- start 2023-09-29T23:15:07.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 1 -- start start 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 10 monclient: init 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 10 monclient: _add_conns ranks=[7,3,9,5,10,2,4,1,6,0,8] 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 0x7f080006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 10 monclient(hunting): picked mon.noname-i con 0x7f08000a03a0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.327+0000 7f0805803700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 0x7f080006f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): picked mon.noname-g con 0x7f080006eda0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080006f6b0 0x7f080009db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): picked mon.noname-a con 0x7f080006f6b0 addr v2:172.21.15.12:3300/0 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000a0770 con 0x7f080006f6b0 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080006ca50 con 0x7f080006eda0 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f080006e950 con 0x7f08000a03a0 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient(hunting): authenticate will time out at 1216.688053s 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080006f6b0 0x7f080009db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f080006f6b0 0x7f080009db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49446/0 (socket says 172.21.15.12:49446) 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 -- 172.21.15.12:0/798672191 learned_addr learned my addr 172.21.15.12:0/798672191 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): get_auth_request con 0x7f080006f6b0 auth_method 0 2023-09-29T23:15:07.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 0x7f080006f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 0x7f080006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 10 monclient(hunting): get_auth_request con 0x7f08000a03a0 auth_method 0 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 10 monclient(hunting): get_auth_request con 0x7f080006eda0 auth_method 0 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_done global_id 4581 payload 274 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 -- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 msgr2=0x7f080006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 0x7f080006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 -- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 msgr2=0x7f080006f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 0x7f080006f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 monclient: found mon.noname-a 2023-09-29T23:15:07.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 -- 172.21.15.12:0/798672191 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800068ed0 con 0x7f080006f6b0 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07feffd700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 0x7f080006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fe7fc700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 0x7f080006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080006f6b0 0x7f080009db80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f07f80099b0 tx=0x7f07f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da8d0507569a1a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 1 -- 172.21.15.12:0/798672191 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f07f800bce0 con 0x7f080006f6b0 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: dump: 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:07.341 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:07.342 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 5 monclient: authenticate success, global_id 4581 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 1 -- 172.21.15.12:0/798672191 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f800be60 con 0x7f080006f6b0 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f0805803700 4 set_mon_vals no callback set 2023-09-29T23:15:07.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 1 -- 172.21.15.12:0/798672191 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f07f80118a0 con 0x7f080006f6b0 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.328+0000 7f07fdffb700 10 monclient: dump: 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:07.344 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 10 monclient: shutdown 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 -- 172.21.15.12:0/798672191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080006f6b0 msgr2=0x7f080009db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:07.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080006f6b0 0x7f080009db80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f07f80099b0 tx=0x7f07f8009250 comp rx=0 tx=0).stop 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 -- 172.21.15.12:0/798672191 shutdown_connections 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f080006f6b0 0x7f080009db80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f08000a03a0 0x7f080006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 --2- 172.21.15.12:0/798672191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f080006eda0 0x7f080006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 -- 172.21.15.12:0/798672191 >> 172.21.15.12:0/798672191 conn(0x7f080006a9b0 msgr2=0x7f080006ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 -- 172.21.15.12:0/798672191 shutdown_connections 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.329+0000 7f0805803700 1 -- 172.21.15.12:0/798672191 wait complete. 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.330+0000 7f0805803700 10 monclient: build_initial_monmap 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.330+0000 7f0805803700 10 monclient: monmap: 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:07.346 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:07.347 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.330+0000 7f0805803700 1 Processor -- start 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.330+0000 7f0805803700 1 -- start start 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.330+0000 7f0805803700 10 monclient: init 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient: _add_conns ranks=[5,2,0,8,1,4,7,3,6] 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f080006eda0 0x7f0800130fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): picked mon.noname-h con 0x7f080006eda0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): picked mon.noname-b con 0x7f080006f6b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:07.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 0x7f0800140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): picked mon.noname-a con 0x7f08000a03a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080006cc80 con 0x7f08000a03a0 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0800130350 con 0x7f080006f6b0 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0800130070 con 0x7f080006eda0 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f0805803700 10 monclient(hunting): authenticate will time out at 1216.691206s 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 0x7f0800140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54034/0 (socket says 172.21.15.12:54034) 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 1 -- 172.21.15.12:0/925010345 learned_addr learned my addr 172.21.15.12:0/925010345 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): get_auth_request con 0x7f080006f6b0 auth_method 0 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): get_auth_request con 0x7f08000a03a0 auth_method 0 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:07.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07feffd700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f080006eda0 0x7f0800130fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07feffd700 10 monclient(hunting): get_auth_request con 0x7f080006eda0 auth_method 0 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07feffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07feffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient(hunting): handle_auth_done global_id 4590 payload 824 2023-09-29T23:15:07.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 -- 172.21.15.12:0/925010345 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f080006eda0 msgr2=0x7f0800130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f080006eda0 0x7f0800130fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 -- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 msgr2=0x7f0800140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 monclient: found mon.noname-a 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07ff7fe700 1 -- 172.21.15.12:0/925010345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80094f0 con 0x7f08000a03a0 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.331+0000 7f07fe7fc700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07ff7fe700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 0x7f0800140a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08000a0370 tx=0x7f07f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:07.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f07f8006a90 con 0x7f08000a03a0 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: dump: 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:07.353 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f801f070 con 0x7f08000a03a0 2023-09-29T23:15:07.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f07f8026d80 con 0x7f08000a03a0 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07f6ffd700 10 monclient: dump: 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:07.355 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 5 monclient: authenticate success, global_id 4590 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 10 monclient: _renew_subs 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080004c960 con 0x7f08000a03a0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f07fd7fa700 4 set_mon_vals no callback set 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 10 monclient: _renew_subs 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.332+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080012e260 con 0x7f08000a03a0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.333+0000 7f0805803700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.333+0000 7f0805803700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.333+0000 7f0805803700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.333+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0800070bc0 con 0x7f08000a03a0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f07f8026480 con 0x7f08000a03a0 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07f6ffd700 1 --2- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 0x7f07e0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f07f8021070 con 0x7f08000a03a0 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07feffd700 1 --2- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 0x7f07e0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07feffd700 10 monclient: get_auth_request con 0x7f07e0036880 auth_method 0 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.334+0000 7f07feffd700 1 --2- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 0x7f07e0038d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007620 tx=0x7f07f00098b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.339+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07f8026730 con 0x7f08000a03a0 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.339+0000 7f07f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.339+0000 7f07f6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:07.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.579+0000 7f0805803700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:07.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.579+0000 7f0805803700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:07.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.579+0000 7f0805803700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:07.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.579+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f080006d000 con 0x7f08000a03a0 2023-09-29T23:15:07.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.580+0000 7f07f6ffd700 1 -- 172.21.15.12:0/925010345 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f07f8015030 con 0x7f08000a03a0 2023-09-29T23:15:07.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.580+0000 7f07f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:07.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.580+0000 7f07f6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:07.580 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:07.581 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:07.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 msgr2=0x7f07e0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:07.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 0x7f07e0038d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007620 tx=0x7f07f00098b0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 10 monclient: shutdown 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 msgr2=0x7f0800140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 0x7f0800140a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08000a0370 tx=0x7f07f8017000 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 shutdown_connections 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f08000a03a0 0x7f0800140a60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f080006eda0 0x7f0800130fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> v2:172.21.15.153:6812/83257 conn(0x7f07e0036880 0x7f07e0038d30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 --2- 172.21.15.12:0/925010345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f080006f6b0 0x7f0800140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 >> 172.21.15.12:0/925010345 conn(0x7f080006a9b0 msgr2=0x7f080006a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:07.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 shutdown_connections 2023-09-29T23:15:07.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:07.582+0000 7f0805803700 1 -- 172.21.15.12:0/925010345 wait complete. 2023-09-29T23:15:10.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 10 monclient: get_monmap_and_config 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 10 monclient: build_initial_monmap 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 10 monclient: monmap: 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:10.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:10.726910+0000 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:10.726910+0000 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:10.727 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:10.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 1 Processor -- start 2023-09-29T23:15:10.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 1 -- start start 2023-09-29T23:15:10.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.726+0000 7f151303c700 10 monclient: init 2023-09-29T23:15:10.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient: _add_conns ranks=[6,1,3,10,4,2,8,0,7,9,5] 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): picked mon.noname-d con 0x7f150c099660 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f150c096cd0 addr v1:172.21.15.153:6789/0 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150c09a140 0x7f150c137570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): picked mon.noname-g con 0x7f150c09a140 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c068ed0 con 0x7f150c096cd0 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c09ba80 con 0x7f150c09a140 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:10.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f150c09a970 con 0x7f150c099660 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f151303c700 10 monclient(hunting): authenticate will time out at 1220.087374s 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:50572/0 (socket says 172.21.15.12:50572) 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 -- 172.21.15.12:0/1374187356 learned_addr learned my addr 172.21.15.12:0/1374187356 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): get_auth_request con 0x7f150c099660 auth_method 0 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:10.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f15115d9700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150c09a140 0x7f150c137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f15115d9700 10 monclient(hunting): get_auth_request con 0x7f150c09a140 auth_method 0 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f15115d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f15115d9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f15115d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient(hunting): handle_auth_done global_id 4551 payload 274 2023-09-29T23:15:10.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 -- 172.21.15.12:0/1374187356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150c09a140 msgr2=0x7f150c137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150c09a140 0x7f150c137570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 -- 172.21.15.12:0/1374187356 >> v1:172.21.15.153:6789/0 conn(0x7f150c096cd0 legacy=0x7f150c099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient: _finish_hunting 0 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 monclient: found mon.noname-d 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.727+0000 7f1510dd8700 1 -- 172.21.15.12:0/1374187356 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c04c960 con 0x7f150c099660 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f1510dd8700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1500009a10 tx=0x7f1500009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=bfbb55d94917ef3b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 1 -- 172.21.15.12:0/1374187356 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f150000bc60 con 0x7f150c099660 2023-09-29T23:15:10.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: dump: 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:10.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:10.742 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: _finish_auth 0 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 1 -- 172.21.15.12:0/1374187356 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150000bde0 con 0x7f150c099660 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 5 monclient: authenticate success, global_id 4551 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 1 -- 172.21.15.12:0/1374187356 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f15000118a0 con 0x7f150c099660 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:10.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 4 set_mon_vals no callback set 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f150b7fe700 10 monclient: dump: 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:10.744 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 10 monclient: shutdown 2023-09-29T23:15:10.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 1 -- 172.21.15.12:0/1374187356 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 msgr2=0x7f150c096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.728+0000 7f151303c700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1500009a10 tx=0x7f1500009250 comp rx=0 tx=0).stop 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 -- 172.21.15.12:0/1374187356 shutdown_connections 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150c09a140 0x7f150c137570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 --2- 172.21.15.12:0/1374187356 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f150c099660 0x7f150c096790 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 -- 172.21.15.12:0/1374187356 >> 172.21.15.12:0/1374187356 conn(0x7f150c09ee30 msgr2=0x7f150c09f230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 -- 172.21.15.12:0/1374187356 shutdown_connections 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 -- 172.21.15.12:0/1374187356 wait complete. 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 10 monclient: build_initial_monmap 2023-09-29T23:15:10.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 10 monclient: monmap: 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:10.747 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.729+0000 7f151303c700 1 Processor -- start 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 -- start start 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient: init 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient: _add_conns ranks=[0,7,5,1,8,3,2,6,4] 2023-09-29T23:15:10.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): picked mon.noname-a con 0x7f150c096cd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f150c099660 0x7f150c142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): picked mon.noname-i con 0x7f150c099660 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f150c09a140 0x7f150c08e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): picked mon.noname-h con 0x7f150c09a140 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c09afc0 con 0x7f150c096cd0 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f150c06f460 con 0x7f150c09a140 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f150c093320 con 0x7f150c099660 2023-09-29T23:15:10.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f151303c700 10 monclient(hunting): authenticate will time out at 1220.090621s 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49464/0 (socket says 172.21.15.12:49464) 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 1 -- 172.21.15.12:0/137120354 learned_addr learned my addr 172.21.15.12:0/137120354 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): get_auth_request con 0x7f150c096cd0 auth_method 0 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f1510dd8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f150bfff700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f150c099660 0x7f150c142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f15115d9700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f150c09a140 0x7f150c08e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f150bfff700 10 monclient(hunting): get_auth_request con 0x7f150c099660 auth_method 0 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f150bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f150bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f150bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f15115d9700 10 monclient(hunting): get_auth_request con 0x7f150c09a140 auth_method 0 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f15115d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f15115d9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.730+0000 7f15115d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f15115d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f15115d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f15115d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f150bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f150bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f150bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 -- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f150c099660 msgr2=0x7f150c142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f150c099660 0x7f150c142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 -- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f150c09a140 msgr2=0x7f150c08e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f150c09a140 0x7f150c08e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 10 monclient: _finish_hunting 0 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 monclient: found mon.noname-a 2023-09-29T23:15:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 -- 172.21.15.12:0/137120354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000094f0 con 0x7f150c096cd0 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1510dd8700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f150c09c230 tx=0x7f1500016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1500009250 con 0x7f150c096cd0 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: dump: 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.753 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: _finish_auth 0 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1500020070 con 0x7f150c096cd0 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 5 monclient: authenticate success, global_id 4599 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1500006b30 con 0x7f150c096cd0 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 10 monclient: _renew_subs 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c091df0 con 0x7f150c096cd0 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f150affd700 4 set_mon_vals no callback set 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:15:10.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f1508ff9700 10 monclient: dump: 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:10.756 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 10 monclient: _renew_subs 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.731+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c091230 con 0x7f150c096cd0 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1500022030 con 0x7f150c096cd0 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f151303c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f151303c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f151303c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:10.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c0925a0 con 0x7f150c096cd0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f1508ff9700 1 --2- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 0x7f14f0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1500048e00 con 0x7f150c096cd0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f150bfff700 1 --2- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 0x7f14f0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.732+0000 7f150bfff700 10 monclient: get_auth_request con 0x7f14f0036880 auth_method 0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.733+0000 7f150bfff700 1 --2- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 0x7f14f0038d30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f150c069d20 tx=0x7f14f8007ec0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.737+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1500006cb0 con 0x7f150c096cd0 2023-09-29T23:15:10.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.737+0000 7f1508ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:10.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.737+0000 7f1508ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:10.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f151303c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:10.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f151303c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f151303c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f150c0925a0 con 0x7f150c096cd0 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f1508ff9700 1 -- 172.21.15.12:0/137120354 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f15000193f0 con 0x7f150c096cd0 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f1508ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.974+0000 7f1508ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:10.975 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 msgr2=0x7f14f0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 0x7f14f0038d30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f150c069d20 tx=0x7f14f8007ec0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 10 monclient: shutdown 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 msgr2=0x7f150c1426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f150c09c230 tx=0x7f1500016000 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 shutdown_connections 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> v2:172.21.15.153:6812/83257 conn(0x7f14f0036880 0x7f14f0038d30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f150c096cd0 0x7f150c1426c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f150c09a140 0x7f150c08e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 --2- 172.21.15.12:0/137120354 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f150c099660 0x7f150c142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 >> 172.21.15.12:0/137120354 conn(0x7f150c09ee30 msgr2=0x7f150c0a1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:10.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 shutdown_connections 2023-09-29T23:15:10.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:10.976+0000 7f151303c700 1 -- 172.21.15.12:0/137120354 wait complete. 2023-09-29T23:15:13.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.116+0000 7f566e18f700 10 monclient: get_monmap_and_config 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.116+0000 7f566e18f700 10 monclient: build_initial_monmap 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.116+0000 7f566e18f700 10 monclient: monmap: 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:14.117057+0000 2023-09-29T23:15:14.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:14.117057+0000 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:14.117 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:14.118 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:14.118 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.116+0000 7f566e18f700 1 Processor -- start 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 1 -- start start 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient: init 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient: _add_conns ranks=[1,6,0,3,4,10,8,2,7,5,9] 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f56680a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:15:14.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 0x7f56680a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-d con 0x7f56680a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5668138980 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5668068ed0 con 0x7f5668138980 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566804c960 con 0x7f56680a1f90 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f566809e350 con 0x7f56680a2360 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f566e18f700 10 monclient(hunting): authenticate will time out at 1223.477514s 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5667fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5668138980 0x7f566813ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48112/0 (socket says 172.21.15.12:48112) 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5667fff700 1 -- 172.21.15.12:0/3223788826 learned_addr learned my addr 172.21.15.12:0/3223788826 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:15:14.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 0x7f56680a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5666ffd700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 msgr2=0x7f56680a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5666ffd700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 msgr2=0x7f56680a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 0x7f56680a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 0x7f56680a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301552480 0 0) 0x7f5668068ed0 con 0x7f5668138980 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f56667fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f56667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f56667fc700 10 monclient(hunting): my global_id is 4608 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.117+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5650003570 con 0x7f5668138980 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571661769 0 0) 0x7f566804c960 con 0x7f56680a1f90 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): my global_id is 4573 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5668068ed0 con 0x7f56680a1f90 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573810847 0 0) 0x7f5650003570 con 0x7f5668138980 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 msgr2=0x7f56680a27b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f56680a2360 0x7f56680a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 >> v1:172.21.15.153:6789/0 conn(0x7f56680a1f90 legacy=0x7f566809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: _finish_hunting 0 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566809da70 con 0x7f5668138980 2023-09-29T23:15:14.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: _finish_auth 0 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f5660001c70 con 0x7f5668138980 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f566e18f700 5 monclient: authenticate success, global_id 4608 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f566e18f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: dump: 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:14.133 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 >> v1:172.21.15.12:6789/0 conn(0x7f5668138980 legacy=0x7f566813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient: _add_conns ranks=[7,8,4,0,5,6,3,1,2] 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5650004810 0x7f5650004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): picked mon.i con 0x7f5650004810 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 0x7f5650009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): picked mon.e con 0x7f5650005120 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650009f90 0x7f565000c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): picked mon.c con 0x7f5650009f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f566809e350 con 0x7f5650009f90 2023-09-29T23:15:14.135 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f565000cbb0 con 0x7f5650004810 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f565000cd20 con 0x7f5650005120 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56667fc700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f566e18f700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 0x7f5650009a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): get_auth_request con 0x7f5650005120 auth_method 0 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650009f90 0x7f565000c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5650004810 0x7f5650004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): get_auth_request con 0x7f5650009f90 auth_method 0 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): get_auth_request con 0x7f5650004810 auth_method 0 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:15:14.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient(hunting): handle_auth_done global_id 4608 payload 293 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5650004810 msgr2=0x7f5650004be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5650004810 0x7f5650004be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650009f90 msgr2=0x7f565000c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650009f90 0x7f565000c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:15:14.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 monclient: found mon.e 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 -- 172.21.15.12:0/3223788826 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650003570 con 0x7f5650005120 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650009f90 0x7f565000c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5667fff700 1 -- 172.21.15.12:0/3223788826 reap_dead start 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.118+0000 7f5666ffd700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 0x7f5650009a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f565c006bc0 tx=0x7f565c008040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ac20b489fdd98ba4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f565c00a470 con 0x7f5650005120 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:14.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: dump: 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:14.140 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c00ca80 con 0x7f5650005120 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 1 -- 172.21.15.12:0/3223788826 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f565c00cbf0 con 0x7f5650005120 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f56667fc700 10 monclient: dump: 2023-09-29T23:15:14.141 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:14.142 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 4 set_mon_vals no callback set 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 10 monclient: shutdown 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 -- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 msgr2=0x7f5650009a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 0x7f5650009a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f565c006bc0 tx=0x7f565c008040 comp rx=0 tx=0).stop 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 -- 172.21.15.12:0/3223788826 shutdown_connections 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 --2- 172.21.15.12:0/3223788826 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5650005120 0x7f5650009a50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 -- 172.21.15.12:0/3223788826 >> 172.21.15.12:0/3223788826 conn(0x7f56680a3b50 msgr2=0x7f56680a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 -- 172.21.15.12:0/3223788826 shutdown_connections 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.119+0000 7f566e18f700 1 -- 172.21.15.12:0/3223788826 wait complete. 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.120+0000 7f566e18f700 10 monclient: build_initial_monmap 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.120+0000 7f566e18f700 10 monclient: monmap: 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.120+0000 7f566e18f700 1 Processor -- start 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.120+0000 7f566e18f700 1 -- start start 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.120+0000 7f566e18f700 10 monclient: init 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient: _add_conns ranks=[4,1,5,3,6,7,0,8,2] 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-c con 0x7f5650005120 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 0x7f5668097a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-f con 0x7f5668142710 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5668097f60 0x7f566808c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): picked mon.noname-h con 0x7f5668097f60 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566809ffc0 con 0x7f5668142710 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f566809e510 con 0x7f5650005120 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f566808ed00 con 0x7f5668097f60 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f566e18f700 10 monclient(hunting): authenticate will time out at 1223.481322s 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:35474/0 (socket says 172.21.15.12:35474) 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 -- 172.21.15.12:0/626339264 learned_addr learned my addr 172.21.15.12:0/626339264 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): get_auth_request con 0x7f5650005120 auth_method 0 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:14.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 0x7f5668097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 10 monclient(hunting): get_auth_request con 0x7f5668142710 auth_method 0 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5667fff700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5668097f60 0x7f566808c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5667fff700 10 monclient(hunting): get_auth_request con 0x7f5668097f60 auth_method 0 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5667fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5667fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5667fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient(hunting): handle_auth_done global_id 4531 payload 824 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 -- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5668097f60 msgr2=0x7f566808c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5668097f60 0x7f566808c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 -- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 msgr2=0x7f5668097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 0x7f5668097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 monclient: found mon.noname-c 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 -- 172.21.15.12:0/626339264 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566808fbf0 con 0x7f5650005120 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f5666ffd700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 0x7f5668097a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f56677fe700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5658002b30 tx=0x7f5658007d30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.121+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5658003460 con 0x7f5650005120 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:14.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: dump: 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:14.150 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: _finish_auth 0 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5658007050 con 0x7f5650005120 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 5 monclient: authenticate success, global_id 4531 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 10 monclient: _renew_subs 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:14.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566808fed0 con 0x7f5650005120 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5658007640 con 0x7f5650005120 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 10 monclient: dump: 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:14.152 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 10 monclient: _renew_subs 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f5665ffb700 4 set_mon_vals no callback set 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:14.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566808e0e0 con 0x7f5650005120 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f56580071c0 con 0x7f5650005120 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.122+0000 7f566e18f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f566e18f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f566e18f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f564f7fe700 1 --2- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 0x7f5654038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5668090060 con 0x7f5650005120 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5658047590 con 0x7f5650005120 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f5666ffd700 1 --2- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 0x7f5654038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f5666ffd700 10 monclient: get_auth_request con 0x7f5654036850 auth_method 0 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.123+0000 7f5666ffd700 1 --2- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 0x7f5654038d00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f565c000c00 tx=0x7f565c019000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:14.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.127+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5658012d90 con 0x7f5650005120 2023-09-29T23:15:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.128+0000 7f564f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.128+0000 7f564f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.363+0000 7f566e18f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.363+0000 7f566e18f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.363+0000 7f566e18f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.363+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5668070bc0 con 0x7f5650005120 2023-09-29T23:15:14.364 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.364+0000 7f564f7fe700 1 -- 172.21.15.12:0/626339264 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5658012350 con 0x7f5650005120 2023-09-29T23:15:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.364+0000 7f564f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:14.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.364+0000 7f564f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:14.365 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:14.365 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 msgr2=0x7f5654038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 0x7f5654038d00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f565c000c00 tx=0x7f565c019000 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 10 monclient: shutdown 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 msgr2=0x7f56681421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5658002b30 tx=0x7f5658007d30 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 shutdown_connections 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> v2:172.21.15.153:6812/83257 conn(0x7f5654036850 0x7f5654038d00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5668097f60 0x7f566808c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5650005120 0x7f56681421d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 --2- 172.21.15.12:0/626339264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5668142710 0x7f5668097a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 >> 172.21.15.12:0/626339264 conn(0x7f56680a3b50 msgr2=0x7f566809a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 shutdown_connections 2023-09-29T23:15:14.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:14.366+0000 7f566e18f700 1 -- 172.21.15.12:0/626339264 wait complete. 2023-09-29T23:15:17.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.502+0000 7f1c44ffa700 10 monclient: get_monmap_and_config 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.502+0000 7f1c44ffa700 10 monclient: build_initial_monmap 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.502+0000 7f1c44ffa700 10 monclient: monmap: 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:17.503 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:17.503669+0000 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:17.503669+0000 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:17.504 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:17.505 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:17.505 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:17.505 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:17.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.503+0000 7f1c44ffa700 1 Processor -- start 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.503+0000 7f1c44ffa700 1 -- start start 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.503+0000 7f1c44ffa700 10 monclient: init 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient: _add_conns ranks=[5,6,0,4,1,7,3,10,9,8,2] 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c40133130 0x7f1c400a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-h con 0x7f1c40133130 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 0x7f1c40133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-d con 0x7f1c40133500 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c4012fc20 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:17.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c40068ed0 con 0x7f1c4012fc20 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1c400a27a0 con 0x7f1c40133130 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4009adf0 con 0x7f1c40133500 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c44ffa700 10 monclient(hunting): authenticate will time out at 1226.864250s 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3ffff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1c4012fc20 0x7f1c40130090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48126/0 (socket says 172.21.15.12:48126) 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3ffff700 1 -- 172.21.15.12:0/3642876017 learned_addr learned my addr 172.21.15.12:0/3642876017 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 0x7f1c40133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): get_auth_request con 0x7f1c40133500 auth_method 0 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3f7fe700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c40133130 0x7f1c400a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3f7fe700 10 monclient(hunting): get_auth_request con 0x7f1c40133130 auth_method 0 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:17.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3e7fc700 1 -- 172.21.15.12:0/3642876017 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170413149 0 0) 0x7f1c40068ed0 con 0x7f1c4012fc20 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3e7fc700 10 monclient(hunting): my global_id is 4617 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3e7fc700 1 -- 172.21.15.12:0/3642876017 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c28003570 con 0x7f1c4012fc20 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient(hunting): handle_auth_done global_id 4560 payload 274 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 -- 172.21.15.12:0/3642876017 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c40133130 msgr2=0x7f1c400a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c40133130 0x7f1c400a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 -- 172.21.15.12:0/3642876017 >> v1:172.21.15.12:6789/0 conn(0x7f1c4012fc20 legacy=0x7f1c40130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient: _finish_hunting 0 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 monclient: found mon.noname-d 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:17.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 -- 172.21.15.12:0/3642876017 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4004c960 con 0x7f1c40133500 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.504+0000 7f1c3effd700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 0x7f1c40133950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009910 tx=0x7f1c340091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=70c395902e13a789 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 1 -- 172.21.15.12:0/3642876017 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c3400bb00 con 0x7f1c40133500 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: dump: 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:17.518 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 1 -- 172.21.15.12:0/3642876017 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3400bc80 con 0x7f1c40133500 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 5 monclient: authenticate success, global_id 4560 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 1 -- 172.21.15.12:0/3642876017 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c3400bdf0 con 0x7f1c40133500 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:17.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c3e7fc700 10 monclient: dump: 2023-09-29T23:15:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.521 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:17.522 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 4 set_mon_vals no callback set 2023-09-29T23:15:17.523 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 10 monclient: shutdown 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 -- 172.21.15.12:0/3642876017 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 msgr2=0x7f1c40133950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 0x7f1c40133950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009910 tx=0x7f1c340091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 -- 172.21.15.12:0/3642876017 shutdown_connections 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1c40133130 0x7f1c400a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/3642876017 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1c40133500 0x7f1c40133950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 -- 172.21.15.12:0/3642876017 >> 172.21.15.12:0/3642876017 conn(0x7f1c400a6490 msgr2=0x7f1c400a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 -- 172.21.15.12:0/3642876017 shutdown_connections 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.505+0000 7f1c44ffa700 1 -- 172.21.15.12:0/3642876017 wait complete. 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.506+0000 7f1c44ffa700 10 monclient: build_initial_monmap 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.506+0000 7f1c44ffa700 10 monclient: monmap: 2023-09-29T23:15:17.524 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:17.525 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.506+0000 7f1c44ffa700 1 Processor -- start 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.506+0000 7f1c44ffa700 1 -- start start 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.506+0000 7f1c44ffa700 10 monclient: init 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient: _add_conns ranks=[1,8,7,6,3,4,2,0,5] 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4012fc20 0x7f1c401423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-f con 0x7f1c4012fc20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:17.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-e con 0x7f1c40133130 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): picked mon.noname-i con 0x7f1c40133500 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4011efd0 con 0x7f1c4012fc20 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4009b020 con 0x7f1c40133500 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4011ea10 con 0x7f1c40133130 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c44ffa700 10 monclient(hunting): authenticate will time out at 1226.867309s 2023-09-29T23:15:17.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:60742/0 (socket says 172.21.15.12:60742) 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 -- 172.21.15.12:0/971004470 learned_addr learned my addr 172.21.15.12:0/971004470 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): get_auth_request con 0x7f1c40133130 auth_method 0 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:17.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3ffff700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4012fc20 0x7f1c401423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): get_auth_request con 0x7f1c4012fc20 auth_method 0 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3ffff700 1 -- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 msgr2=0x7f1c4006e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3ffff700 1 -- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 msgr2=0x7f1c4006e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3ffff700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3ffff700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:17.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 -- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 msgr2=0x7f1c4006e190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 -- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4012fc20 msgr2=0x7f1c401423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4012fc20 0x7f1c401423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient: _finish_hunting 0 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 monclient: found mon.noname-e 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 -- 172.21.15.12:0/971004470 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c34009420 con 0x7f1c40133130 2023-09-29T23:15:17.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c3effd700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c34013000 tx=0x7f1c34017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.507+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c34006ad0 con 0x7f1c40133130 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: dump: 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:17.531 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: _finish_auth 0 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3401cbf0 con 0x7f1c40133130 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 5 monclient: authenticate success, global_id 4436 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 10 monclient: _renew_subs 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4011ecf0 con 0x7f1c40133130 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c3401cd60 con 0x7f1c40133130 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c3dffb700 4 set_mon_vals no callback set 2023-09-29T23:15:17.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c277fe700 10 monclient: dump: 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:17.534 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 10 monclient: _renew_subs 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.508+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4011ce70 con 0x7f1c40133130 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1c3401a430 con 0x7f1c40133130 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c44ffa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c44ffa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:17.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c44ffa700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c40070bc0 con 0x7f1c40133130 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c277fe700 1 --2- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 0x7f1c2c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1c34048da0 con 0x7f1c40133130 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c3f7fe700 1 --2- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 0x7f1c2c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c3f7fe700 10 monclient: get_auth_request con 0x7f1c2c036850 auth_method 0 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.509+0000 7f1c3f7fe700 1 --2- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 0x7f1c2c038d00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c30006fd0 tx=0x7f1c30008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.513+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c34010560 con 0x7f1c40133130 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.514+0000 7f1c277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.514+0000 7f1c277fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c44ffa700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c44ffa700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c44ffa700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1c4011f1f0 con 0x7f1c40133130 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c277fe700 1 -- 172.21.15.12:0/971004470 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1c34015030 con 0x7f1c40133130 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:17.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.749+0000 7f1c277fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:17.751 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:17.751 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:17.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 msgr2=0x7f1c2c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:17.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 0x7f1c2c038d00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c30006fd0 tx=0x7f1c30008040 comp rx=0 tx=0).stop 2023-09-29T23:15:17.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 10 monclient: shutdown 2023-09-29T23:15:17.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:17.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 msgr2=0x7f1c40142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c34013000 tx=0x7f1c34017000 comp rx=0 tx=0).stop 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 shutdown_connections 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> v2:172.21.15.153:6812/83257 conn(0x7f1c2c036850 0x7f1c2c038d00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1c40133500 0x7f1c4006e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c4012fc20 0x7f1c401423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 --2- 172.21.15.12:0/971004470 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1c40133130 0x7f1c40142930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:17.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 >> 172.21.15.12:0/971004470 conn(0x7f1c400a6490 msgr2=0x7f1c400a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:17.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 shutdown_connections 2023-09-29T23:15:17.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:17.752+0000 7f1c44ffa700 1 -- 172.21.15.12:0/971004470 wait complete. 2023-09-29T23:15:20.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 10 monclient: get_monmap_and_config 2023-09-29T23:15:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 10 monclient: build_initial_monmap 2023-09-29T23:15:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 10 monclient: monmap: 2023-09-29T23:15:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:20.896 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:20.897053+0000 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:20.897053+0000 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:20.897 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:20.898 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 1 Processor -- start 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 1 -- start start 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.896+0000 7f8ced949700 10 monclient: init 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient: _add_conns ranks=[5,6,10,4,7,1,8,2,3,0,9] 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8ce8130fe0 0x7f8ce8131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-h con 0x7f8ce8130fe0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-d con 0x7f8ce806f0b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8ce80969e0 0x7f8ce8096e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-f con 0x7f8ce80969e0 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce81313b0 con 0x7f8ce80969e0 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce8131e00 con 0x7f8ce8130fe0 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce806d690 con 0x7f8ce806f0b0 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:20.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ced949700 10 monclient(hunting): authenticate will time out at 1230.257328s 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:39646/0 (socket says 172.21.15.12:39646) 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 -- 172.21.15.12:0/3767257593 learned_addr learned my addr 172.21.15.12:0/3767257593 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): get_auth_request con 0x7f8ce806f0b0 auth_method 0 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:20.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce6ffd700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8ce8130fe0 0x7f8ce8131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce77fe700 1 --2- 172.21.15.12:0/3767257593 >> v2:172.21.15.153:3300/0 conn(0x7f8ce80969e0 0x7f8ce8096e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce77fe700 10 monclient(hunting): get_auth_request con 0x7f8ce80969e0 auth_method 0 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce6ffd700 10 monclient(hunting): get_auth_request con 0x7f8ce8130fe0 auth_method 0 2023-09-29T23:15:20.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 -- 172.21.15.12:0/3767257593 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8ce8130fe0 msgr2=0x7f8ce8131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8ce8130fe0 0x7f8ce8131740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 -- 172.21.15.12:0/3767257593 >> v2:172.21.15.153:3300/0 conn(0x7f8ce80969e0 msgr2=0x7f8ce8096e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/3767257593 >> v2:172.21.15.153:3300/0 conn(0x7f8ce80969e0 0x7f8ce8096e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient: _finish_hunting 0 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 monclient: found mon.noname-d 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.897+0000 7f8ce67fc700 1 -- 172.21.15.12:0/3767257593 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8068ed0 con 0x7f8ce806f0b0 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00099b0 tx=0x7f8cd0009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4039e464b09cbb07 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 1 -- 172.21.15.12:0/3767257593 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8cd000bce0 con 0x7f8ce806f0b0 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: dump: 2023-09-29T23:15:20.913 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:20.914 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 1 -- 172.21.15.12:0/3767257593 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd000be40 con 0x7f8ce806f0b0 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 5 monclient: authenticate success, global_id 4569 2023-09-29T23:15:20.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 1 -- 172.21.15.12:0/3767257593 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8cd00118a0 con 0x7f8ce806f0b0 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 4 set_mon_vals no callback set 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ce5ffb700 10 monclient: dump: 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:20.916 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:20.917 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 10 monclient: shutdown 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 -- 172.21.15.12:0/3767257593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 msgr2=0x7f8ce806f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00099b0 tx=0x7f8cd0009250 comp rx=0 tx=0).stop 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 -- 172.21.15.12:0/3767257593 shutdown_connections 2023-09-29T23:15:20.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 --2- 172.21.15.12:0/3767257593 >> v2:172.21.15.153:3300/0 conn(0x7f8ce80969e0 0x7f8ce8096e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8ce8130fe0 0x7f8ce8131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 --2- 172.21.15.12:0/3767257593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce806f0b0 0x7f8ce806f500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 -- 172.21.15.12:0/3767257593 >> 172.21.15.12:0/3767257593 conn(0x7f8ce80a4430 msgr2=0x7f8ce80a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 -- 172.21.15.12:0/3767257593 shutdown_connections 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.898+0000 7f8ced949700 1 -- 172.21.15.12:0/3767257593 wait complete. 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.899+0000 7f8ced949700 10 monclient: build_initial_monmap 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.899+0000 7f8ced949700 10 monclient: monmap: 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:20.919 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:20.920 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.899+0000 7f8ced949700 1 Processor -- start 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.899+0000 7f8ced949700 1 -- start start 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.899+0000 7f8ced949700 10 monclient: init 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:20.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient: _add_conns ranks=[3,6,8,7,5,1,4,0,2] 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ce806f0b0 0x7f8ce8140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-g con 0x7f8ce806f0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-d con 0x7f8ce80969e0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8ce8130fe0 0x7f8ce8098a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): picked mon.noname-e con 0x7f8ce8130fe0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce80a0550 con 0x7f8ce806f0b0 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce80a0180 con 0x7f8ce80969e0 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce811b320 con 0x7f8ce8130fe0 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ced949700 10 monclient(hunting): authenticate will time out at 1230.260416s 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:39656/0 (socket says 172.21.15.12:39656) 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 -- 172.21.15.12:0/2182639079 learned_addr learned my addr 172.21.15.12:0/2182639079 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): get_auth_request con 0x7f8ce80969e0 auth_method 0 2023-09-29T23:15:20.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8ce8130fe0 0x7f8ce8098a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ce806f0b0 0x7f8ce8140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): get_auth_request con 0x7f8ce806f0b0 auth_method 0 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): get_auth_request con 0x7f8ce8130fe0 auth_method 0 2023-09-29T23:15:20.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:20.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient(hunting): handle_auth_done global_id 4578 payload 824 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 -- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8ce8130fe0 msgr2=0x7f8ce8098a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8ce8130fe0 0x7f8ce8098a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 -- 172.21.15.12:0/2182639079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ce806f0b0 msgr2=0x7f8ce8140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ce806f0b0 0x7f8ce8140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient: _finish_hunting 0 2023-09-29T23:15:20.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 monclient: found mon.noname-d 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.900+0000 7f8ce67fc700 1 -- 172.21.15.12:0/2182639079 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00094f0 con 0x7f8ce80969e0 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ce67fc700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ce806d7d0 tx=0x7f8cd0017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8cd0011ab0 con 0x7f8ce80969e0 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: dump: 2023-09-29T23:15:20.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:20.928 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 5 monclient: authenticate success, global_id 4578 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd0011c70 con 0x7f8ce80969e0 2023-09-29T23:15:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 10 monclient: _renew_subs 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce804c960 con 0x7f8ce80969e0 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8cd0011de0 con 0x7f8ce80969e0 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ce57fa700 4 set_mon_vals no callback set 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8cdeffd700 10 monclient: dump: 2023-09-29T23:15:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 10 monclient: _renew_subs 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.901+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce8099fc0 con 0x7f8ce80969e0 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8cd0011630 con 0x7f8ce80969e0 2023-09-29T23:15:20.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ced949700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ced949700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ced949700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce80a4d60 con 0x7f8ce80969e0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8cdeffd700 1 --2- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 0x7f8cc8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8cd004c870 con 0x7f8ce80969e0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ce6ffd700 1 --2- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 0x7f8cc8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.902+0000 7f8ce6ffd700 10 monclient: get_auth_request con 0x7f8cc8036880 auth_method 0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.903+0000 7f8ce6ffd700 1 --2- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 0x7f8cc8038d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006fd0 tx=0x7f8ce0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:20.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.907+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8cd002f940 con 0x7f8ce80969e0 2023-09-29T23:15:20.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.907+0000 7f8cdeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:20.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:20.907+0000 7f8cdeffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.142+0000 7f8ced949700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.142+0000 7f8ced949700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.142+0000 7f8ced949700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.142+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8ce8070bc0 con 0x7f8ce80969e0 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.143+0000 7f8cdeffd700 1 -- 172.21.15.12:0/2182639079 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8cd002c020 con 0x7f8ce80969e0 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.143+0000 7f8cdeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:21.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.143+0000 7f8cdeffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:21.144 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:21.144 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:21.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 msgr2=0x7f8cc8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:21.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 0x7f8cc8038d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006fd0 tx=0x7f8ce0008040 comp rx=0 tx=0).stop 2023-09-29T23:15:21.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 10 monclient: shutdown 2023-09-29T23:15:21.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 msgr2=0x7f8ce80984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ce806d7d0 tx=0x7f8cd0017000 comp rx=0 tx=0).stop 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 shutdown_connections 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f8ce8130fe0 0x7f8ce8098a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> v2:172.21.15.153:6812/83257 conn(0x7f8cc8036880 0x7f8cc8038d30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ce806f0b0 0x7f8ce8140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 --2- 172.21.15.12:0/2182639079 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8ce80969e0 0x7f8ce80984f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:21.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 >> 172.21.15.12:0/2182639079 conn(0x7f8ce80a4430 msgr2=0x7f8ce806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:21.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 shutdown_connections 2023-09-29T23:15:21.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:21.145+0000 7f8ced949700 1 -- 172.21.15.12:0/2182639079 wait complete. 2023-09-29T23:15:24.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:24.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.285+0000 7f0524256700 10 monclient: get_monmap_and_config 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.285+0000 7f0524256700 10 monclient: build_initial_monmap 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.285+0000 7f0524256700 10 monclient: monmap: 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:24.286521+0000 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:24.286521+0000 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:24.286 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:24.287 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.286+0000 7f0524256700 1 Processor -- start 2023-09-29T23:15:24.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.286+0000 7f0524256700 1 -- start start 2023-09-29T23:15:24.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.286+0000 7f0524256700 10 monclient: init 2023-09-29T23:15:24.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.286+0000 7f0524256700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient: _add_conns ranks=[4,5,0,10,8,9,6,3,2,1,7] 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): picked mon.noname-c con 0x7f051c0a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 0x7f051c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): picked mon.noname-h con 0x7f051c0a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f051c133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c068ed0 con 0x7f051c133740 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f051c09e350 con 0x7f051c0a1f90 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f051c09ef70 con 0x7f051c0a2360 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient(hunting): authenticate will time out at 1233.647252s 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:46498/0 (socket says 172.21.15.12:46498) 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f05227f3700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f051c133740 0x7f051c133bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50582/0 (socket says 172.21.15.12:50582) 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 -- 172.21.15.12:0/746584163 learned_addr learned my addr 172.21.15.12:0/746584163 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): get_auth_request con 0x7f051c0a1f90 auth_method 0 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:24.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f05217f1700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 0x7f051c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient(hunting): handle_auth_done global_id 4540 payload 274 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 -- 172.21.15.12:0/746584163 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 msgr2=0x7f051c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 0x7f051c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 -- 172.21.15.12:0/746584163 >> v1:172.21.15.12:6789/0 conn(0x7f051c133740 legacy=0x7f051c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient: _finish_hunting 0 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 monclient: found mon.noname-c 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 1 -- 172.21.15.12:0/746584163 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47579718 0 0) 0x7f051c068ed0 con 0x7f051c133740 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:24.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 -- 172.21.15.12:0/746584163 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c04c960 con 0x7f051c0a1f90 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f05217f1700 10 monclient: get_auth_request con 0x7f051c0a2360 auth_method 0 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f05217f1700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 0x7f051c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0521ff2700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0518009a10 tx=0x7f0518009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b02b874132cfa7a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 1 -- 172.21.15.12:0/746584163 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f051800bca0 con 0x7f051c0a1f90 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: dump: 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.299 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: _finish_auth 0 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 5 monclient: authenticate success, global_id 4540 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 1 -- 172.21.15.12:0/746584163 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051800be20 con 0x7f051c0a1f90 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 1 -- 172.21.15.12:0/746584163 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f05180118a0 con 0x7f051c0a1f90 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0524256700 4 set_mon_vals no callback set 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:24.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.287+0000 7f0520ff0700 10 monclient: dump: 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:24.302 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 10 monclient: shutdown 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 -- 172.21.15.12:0/746584163 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 msgr2=0x7f051c09f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0518009a10 tx=0x7f0518009250 comp rx=0 tx=0).stop 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 -- 172.21.15.12:0/746584163 shutdown_connections 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f051c0a1f90 0x7f051c09f5e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 --2- 172.21.15.12:0/746584163 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f051c0a2360 0x7f051c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 -- 172.21.15.12:0/746584163 >> 172.21.15.12:0/746584163 conn(0x7f051c0a3b50 msgr2=0x7f051c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 -- 172.21.15.12:0/746584163 shutdown_connections 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.288+0000 7f0524256700 1 -- 172.21.15.12:0/746584163 wait complete. 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient: build_initial_monmap 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient: monmap: 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:24.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 Processor -- start 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 -- start start 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient: init 2023-09-29T23:15:24.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient: _add_conns ranks=[6,1,8,0,7,2,4,5,3] 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): picked mon.noname-d con 0x7f051c0a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0a2360 0x7f051c09cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): picked mon.noname-f con 0x7f051c0a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f051c133740 0x7f051c12d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): picked mon.noname-e con 0x7f051c133740 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1335e0 con 0x7f051c0a2360 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f051c12fac0 con 0x7f051c0a1f90 2023-09-29T23:15:24.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1309b0 con 0x7f051c133740 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0524256700 10 monclient(hunting): authenticate will time out at 1233.649850s 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:53514/0 (socket says 172.21.15.12:53514) 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 1 -- 172.21.15.12:0/3784957622 learned_addr learned my addr 172.21.15.12:0/3784957622 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 10 monclient(hunting): get_auth_request con 0x7f051c0a1f90 auth_method 0 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f0521ff2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.289+0000 7f05227f3700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f051c133740 0x7f051c12d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): get_auth_request con 0x7f051c133740 auth_method 0 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05227f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05217f1700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0a2360 0x7f051c09cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05217f1700 10 monclient(hunting): get_auth_request con 0x7f051c0a2360 auth_method 0 2023-09-29T23:15:24.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05217f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05217f1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f05217f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 -- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f051c133740 msgr2=0x7f051c12d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f051c133740 0x7f051c12d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 -- 172.21.15.12:0/3784957622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0a2360 msgr2=0x7f051c09cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0a2360 0x7f051c09cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient: _finish_hunting 0 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 monclient: found mon.noname-d 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:24.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 -- 172.21.15.12:0/3784957622 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05180094f0 con 0x7f051c0a1f90 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0521ff2700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f051c09e7b0 tx=0x7f0518017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0518011ab0 con 0x7f051c0a1f90 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: dump: 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:24.310 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:24.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0518011c70 con 0x7f051c0a1f90 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 5 monclient: authenticate success, global_id 4587 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 10 monclient: _renew_subs 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c130c90 con 0x7f051c0a1f90 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0518011de0 con 0x7f051c0a1f90 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0511ffb700 10 monclient: dump: 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:24.312 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:24.313 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0513fff700 4 set_mon_vals no callback set 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 10 monclient: _renew_subs 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.290+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c12ee80 con 0x7f051c0a1f90 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0518011630 con 0x7f051c0a1f90 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0524256700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0524256700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0524256700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c070bc0 con 0x7f051c0a1f90 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0511ffb700 1 --2- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 0x7f0500038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.291+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0518048b00 con 0x7f051c0a1f90 2023-09-29T23:15:24.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.292+0000 7f05217f1700 1 --2- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 0x7f0500038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:24.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.292+0000 7f05217f1700 10 monclient: get_auth_request con 0x7f0500036850 auth_method 0 2023-09-29T23:15:24.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.292+0000 7f05217f1700 1 --2- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 0x7f0500038d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f051c12d1a0 tx=0x7f050c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:24.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.296+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05180118e0 con 0x7f051c0a1f90 2023-09-29T23:15:24.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.296+0000 7f0511ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:24.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.296+0000 7f0511ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0524256700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0524256700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:24.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0524256700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f051c066790 con 0x7f051c0a1f90 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0511ffb700 1 -- 172.21.15.12:0/3784957622 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0518028070 con 0x7f051c0a1f90 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0511ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.535+0000 7f0511ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:24.536 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 msgr2=0x7f0500038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 0x7f0500038d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f051c12d1a0 tx=0x7f050c008040 comp rx=0 tx=0).stop 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 10 monclient: shutdown 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 msgr2=0x7f051c09c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f051c09e7b0 tx=0x7f0518017000 comp rx=0 tx=0).stop 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 shutdown_connections 2023-09-29T23:15:24.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> v2:172.21.15.153:6812/83257 conn(0x7f0500036850 0x7f0500038d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f051c133740 0x7f051c12d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f051c0a1f90 0x7f051c09c760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 --2- 172.21.15.12:0/3784957622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0a2360 0x7f051c09cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 >> 172.21.15.12:0/3784957622 conn(0x7f051c0a3b50 msgr2=0x7f051c0a6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 shutdown_connections 2023-09-29T23:15:24.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:24.537+0000 7f0524256700 1 -- 172.21.15.12:0/3784957622 wait complete. 2023-09-29T23:15:27.550 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:27.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.680+0000 7fc355885700 10 monclient: get_monmap_and_config 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.680+0000 7fc355885700 10 monclient: build_initial_monmap 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.680+0000 7fc355885700 10 monclient: monmap: 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:27.681704+0000 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:27.681704+0000 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:27.681 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:27.682 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.681+0000 7fc355885700 1 Processor -- start 2023-09-29T23:15:27.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.681+0000 7fc355885700 1 -- start start 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.681+0000 7fc355885700 10 monclient: init 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.681+0000 7fc355885700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.681+0000 7fc355885700 10 monclient: _add_conns ranks=[2,5,6,9,0,7,10,3,1,4,8] 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): picked mon.noname-b con 0x7fc3500a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 0x7fc3500a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): picked mon.noname-h con 0x7fc3500a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): picked mon.noname-d con 0x7fc350133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35009e350 con 0x7fc3500a1f90 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc35009ef70 con 0x7fc3500a2360 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc35009da70 con 0x7fc350133740 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 10 monclient(hunting): authenticate will time out at 1237.042127s 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:53542/0 (socket says 172.21.15.12:53542) 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:43314/0 (socket says 172.21.15.12:43314) 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 -- 172.21.15.12:0/2751090129 learned_addr learned my addr 172.21.15.12:0/2751090129 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:27.692 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): get_auth_request con 0x7fc350133740 auth_method 0 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): get_auth_request con 0x7fc3500a1f90 auth_method 0 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 0x7fc3500a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:27.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 10 monclient(hunting): get_auth_request con 0x7fc3500a2360 auth_method 0 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_done global_id 4596 payload 274 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 -- 172.21.15.12:0/2751090129 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 msgr2=0x7fc3500a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 0x7fc3500a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 -- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 msgr2=0x7fc35009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:27.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 monclient: found mon.noname-d 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 -- 172.21.15.12:0/2751090129 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350068ed0 con 0x7fc350133740 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34e7fc700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 0x7fc3500a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34effd700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34f7fe700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc348009fb0 tx=0x7fc348009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=94ddb5a47c0c134b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 1 -- 172.21.15.12:0/2751090129 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc34800e070 con 0x7fc350133740 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:27.695 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 10 monclient: dump: 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:27.696 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc34dffb700 1 -- 172.21.15.12:0/2751090129 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34800be80 con 0x7fc350133740 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 5 monclient: authenticate success, global_id 4596 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.682+0000 7fc355885700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc34dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:27.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc34dffb700 1 -- 172.21.15.12:0/2751090129 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc3480092e0 con 0x7fc350133740 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 4 set_mon_vals no callback set 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc34dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc34dffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc34dffb700 10 monclient: dump: 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:27.698 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 10 monclient: shutdown 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:27.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 -- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 msgr2=0x7fc350133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc348009fb0 tx=0x7fc348009840 comp rx=0 tx=0).stop 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 -- 172.21.15.12:0/2751090129 shutdown_connections 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fc350133740 0x7fc350133bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc3500a1f90 0x7fc35009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 --2- 172.21.15.12:0/2751090129 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a2360 0x7fc3500a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 -- 172.21.15.12:0/2751090129 >> 172.21.15.12:0/2751090129 conn(0x7fc3500a3b50 msgr2=0x7fc3500a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 -- 172.21.15.12:0/2751090129 shutdown_connections 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.683+0000 7fc355885700 1 -- 172.21.15.12:0/2751090129 wait complete. 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.684+0000 7fc355885700 10 monclient: build_initial_monmap 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.684+0000 7fc355885700 10 monclient: monmap: 2023-09-29T23:15:27.700 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:27.701 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.684+0000 7fc355885700 1 Processor -- start 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.684+0000 7fc355885700 1 -- start start 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.684+0000 7fc355885700 10 monclient: init 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient: _add_conns ranks=[5,4,0,3,6,7,2,1,8] 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 0x7fc3501422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): picked mon.noname-h con 0x7fc3500a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): picked mon.noname-c con 0x7fc3500a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): picked mon.noname-a con 0x7fc350133740 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35012feb0 con 0x7fc350133740 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc350130da0 con 0x7fc3500a2360 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc350131080 con 0x7fc3500a1f90 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient(hunting): authenticate will time out at 1237.045160s 2023-09-29T23:15:27.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59370/0 (socket says 172.21.15.12:59370) 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:46502/0 (socket says 172.21.15.12:46502) 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 1 -- 172.21.15.12:0/4244588819 learned_addr learned my addr 172.21.15.12:0/4244588819 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): get_auth_request con 0x7fc350133740 auth_method 0 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): get_auth_request con 0x7fc3500a2360 auth_method 0 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 0x7fc3501422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 10 monclient(hunting): get_auth_request con 0x7fc3500a1f90 auth_method 0 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient(hunting): handle_auth_done global_id 4549 payload 824 2023-09-29T23:15:27.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 -- 172.21.15.12:0/4244588819 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 msgr2=0x7fc3501422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 0x7fc3501422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 -- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 msgr2=0x7fc35012d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient: _finish_hunting 0 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 monclient: found mon.noname-c 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 -- 172.21.15.12:0/4244588819 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc340009420 con 0x7fc3500a2360 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34effd700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 0x7fc3501422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34f7fe700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:27.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34e7fc700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc35006b460 tx=0x7fc344008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc344013890 con 0x7fc3500a2360 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: dump: 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:27.707 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344013a30 con 0x7fc3500a2360 2023-09-29T23:15:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 5 monclient: authenticate success, global_id 4549 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient: _renew_subs 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc348009ae0 con 0x7fc3500a2360 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc344013ba0 con 0x7fc3500a2360 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc336ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc336ffd700 10 monclient: dump: 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.685+0000 7fc34d7fa700 4 set_mon_vals no callback set 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 10 monclient: _renew_subs 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35004c960 con 0x7fc3500a2360 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc344013d00 con 0x7fc3500a2360 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:27.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35009e520 con 0x7fc3500a2360 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc336ffd700 1 --2- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 0x7fc33c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.686+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc344046f90 con 0x7fc3500a2360 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.687+0000 7fc34effd700 1 --2- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 0x7fc33c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.687+0000 7fc34effd700 10 monclient: get_auth_request con 0x7fc33c036850 auth_method 0 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.687+0000 7fc34effd700 1 --2- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 0x7fc33c038d00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3400093f0 tx=0x7fc340008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:27.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.691+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc34400f040 con 0x7fc3500a2360 2023-09-29T23:15:27.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.691+0000 7fc336ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:27.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.691+0000 7fc336ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc355885700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc355885700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:27.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc355885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc350070bc0 con 0x7fc3500a2360 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc336ffd700 1 -- 172.21.15.12:0/4244588819 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc344010320 con 0x7fc3500a2360 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc336ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.924+0000 7fc336ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:27.925 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 msgr2=0x7fc33c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 0x7fc33c038d00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3400093f0 tx=0x7fc340008040 comp rx=0 tx=0).stop 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 10 monclient: shutdown 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 msgr2=0x7fc350132b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:27.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc35006b460 tx=0x7fc344008040 comp rx=0 tx=0).stop 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 shutdown_connections 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> v2:172.21.15.153:6812/83257 conn(0x7fc33c036850 0x7fc33c038d00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc350133740 0x7fc35012d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fc3500a1f90 0x7fc3501422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.926+0000 7fc355885700 1 --2- 172.21.15.12:0/4244588819 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc3500a2360 0x7fc350132b70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.927+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 >> 172.21.15.12:0/4244588819 conn(0x7fc3500a3b50 msgr2=0x7fc3500a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.927+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 shutdown_connections 2023-09-29T23:15:27.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:27.927+0000 7fc355885700 1 -- 172.21.15.12:0/4244588819 wait complete. 2023-09-29T23:15:30.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:31.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.070+0000 7f4d95127700 10 monclient: get_monmap_and_config 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.070+0000 7f4d95127700 10 monclient: build_initial_monmap 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.070+0000 7f4d95127700 10 monclient: monmap: 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:31.071352+0000 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:31.071352+0000 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:31.071 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 Processor -- start 2023-09-29T23:15:31.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 -- start start 2023-09-29T23:15:31.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient: init 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient: _add_conns ranks=[8,0,5,6,4,7,10,2,3,9,1] 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 0x7f4d9009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-e con 0x7f4d900a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d900a2360 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d90138980 0x7f4d9013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-h con 0x7f4d90138980 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d90068ed0 con 0x7f4d900a2360 2023-09-29T23:15:31.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9009e350 con 0x7f4d90138980 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9009ef70 con 0x7f4d900a1f90 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d95127700 10 monclient(hunting): authenticate will time out at 1240.431623s 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8e59c700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4d900a2360 0x7f4d900a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50584/0 (socket says 172.21.15.12:50584) 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 0x7f4d9009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8e59c700 1 -- 172.21.15.12:0/1744353349 learned_addr learned my addr 172.21.15.12:0/1744353349 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): get_auth_request con 0x7f4d900a1f90 auth_method 0 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8f59e700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d90138980 0x7f4d9013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8f59e700 10 monclient(hunting): get_auth_request con 0x7f4d90138980 auth_method 0 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.071+0000 7f4d8f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 1 -- 172.21.15.12:0/1744353349 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281866211 0 0) 0x7f4d90068ed0 con 0x7f4d900a2360 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient(hunting): my global_id is 4644 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 1 -- 172.21.15.12:0/1744353349 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c003570 con 0x7f4d900a2360 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/1744353349 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d90138980 msgr2=0x7f4d9013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d90138980 0x7f4d9013ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/1744353349 >> v1:172.21.15.12:6789/0 conn(0x7f4d900a2360 legacy=0x7f4d900a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 monclient: found mon.noname-e 2023-09-29T23:15:31.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/1744353349 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9004c960 con 0x7f4d900a1f90 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8ed9d700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 0x7f4d9009f5e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009910 tx=0x7f4d800091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ad514ebfb4cf80d3 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 1 -- 172.21.15.12:0/1744353349 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4d80011ab0 con 0x7f4d900a1f90 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: dump: 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:31.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: _finish_auth 0 2023-09-29T23:15:31.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d95127700 5 monclient: authenticate success, global_id 4463 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 1 -- 172.21.15.12:0/1744353349 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80011c30 con 0x7f4d900a1f90 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d95127700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d95127700 4 set_mon_vals no callback set 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 1 -- 172.21.15.12:0/1744353349 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4d80011da0 con 0x7f4d900a1f90 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.072+0000 7f4d8dd9b700 10 monclient: dump: 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:31.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:31.088 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 10 monclient: shutdown 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 -- 172.21.15.12:0/1744353349 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 msgr2=0x7f4d9009f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 0x7f4d9009f5e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009910 tx=0x7f4d800091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 -- 172.21.15.12:0/1744353349 shutdown_connections 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d90138980 0x7f4d9013ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 --2- 172.21.15.12:0/1744353349 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f4d900a1f90 0x7f4d9009f5e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 -- 172.21.15.12:0/1744353349 >> 172.21.15.12:0/1744353349 conn(0x7f4d900a3b50 msgr2=0x7f4d900a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 -- 172.21.15.12:0/1744353349 shutdown_connections 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 1 -- 172.21.15.12:0/1744353349 wait complete. 2023-09-29T23:15:31.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 10 monclient: build_initial_monmap 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.073+0000 7f4d95127700 10 monclient: monmap: 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:31.090 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 Processor -- start 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 -- start start 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient: init 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient: _add_conns ranks=[4,5,7,2,6,3,0,8,1] 2023-09-29T23:15:31.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-c con 0x7f4d900a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 0x7f4d90097a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-h con 0x7f4d900a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4d90138980 0x7f4d9008d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): picked mon.noname-i con 0x7f4d90138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9008fee0 con 0x7f4d900a1f90 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4d90143040 con 0x7f4d900a2360 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9008eca0 con 0x7f4d90138980 2023-09-29T23:15:31.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d95127700 10 monclient(hunting): authenticate will time out at 1240.434827s 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:46516/0 (socket says 172.21.15.12:46516) 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/3730344495 learned_addr learned my addr 172.21.15.12:0/3730344495 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): get_auth_request con 0x7f4d900a1f90 auth_method 0 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.074+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:31.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 0x7f4d90097a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8f59e700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4d90138980 0x7f4d9008d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 10 monclient(hunting): get_auth_request con 0x7f4d900a2360 auth_method 0 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8f59e700 10 monclient(hunting): get_auth_request con 0x7f4d90138980 auth_method 0 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 10 monclient(hunting): handle_auth_done global_id 4558 payload 824 2023-09-29T23:15:31.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4d90138980 msgr2=0x7f4d9008d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4d90138980 0x7f4d9008d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 msgr2=0x7f4d90097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 0x7f4d90097a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 monclient: found mon.noname-c 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 -- 172.21.15.12:0/3730344495 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80009420 con 0x7f4d900a1f90 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8e59c700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 0x7f4d90097a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8ed9d700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4d80013000 tx=0x7f4d80017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4d8001f070 con 0x7f4d900a1f90 2023-09-29T23:15:31.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: dump: 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:31.096 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8001ae70 con 0x7f4d900a1f90 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 5 monclient: authenticate success, global_id 4558 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 10 monclient: _renew_subs 2023-09-29T23:15:31.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9008fc00 con 0x7f4d900a1f90 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4d80006af0 con 0x7f4d900a1f90 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d8d59a700 4 set_mon_vals no callback set 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d86ffd700 10 monclient: dump: 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:31.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:31.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 10 monclient: _renew_subs 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.075+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9006e430 con 0x7f4d900a1f90 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4d80023070 con 0x7f4d900a1f90 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d95127700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d95127700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d95127700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d90090380 con 0x7f4d900a1f90 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d86ffd700 1 --2- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 0x7f4d70038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4d80048150 con 0x7f4d900a1f90 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d8e59c700 1 --2- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 0x7f4d70038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.076+0000 7f4d8e59c700 10 monclient: get_auth_request con 0x7f4d70036850 auth_method 0 2023-09-29T23:15:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.077+0000 7f4d8e59c700 1 --2- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 0x7f4d70038d00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d780058d0 tx=0x7f4d78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.081+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d8002e770 con 0x7f4d900a1f90 2023-09-29T23:15:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.081+0000 7f4d86ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.081+0000 7f4d86ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d95127700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d95127700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d95127700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d90090020 con 0x7f4d900a1f90 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d86ffd700 1 -- 172.21.15.12:0/3730344495 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4d90090020 con 0x7f4d900a1f90 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d86ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:31.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.319+0000 7f4d86ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:31.321 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:31.321 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 msgr2=0x7f4d70038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 0x7f4d70038d00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d780058d0 tx=0x7f4d78009040 comp rx=0 tx=0).stop 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 10 monclient: shutdown 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 msgr2=0x7f4d9006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4d80013000 tx=0x7f4d80017000 comp rx=0 tx=0).stop 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 shutdown_connections 2023-09-29T23:15:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> v2:172.21.15.153:6812/83257 conn(0x7f4d70036850 0x7f4d70038d00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f4d90138980 0x7f4d9008d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f4d900a1f90 0x7f4d9006dbf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 --2- 172.21.15.12:0/3730344495 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f4d900a2360 0x7f4d90097a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 >> 172.21.15.12:0/3730344495 conn(0x7f4d900a3b50 msgr2=0x7f4d9009a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.321+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 shutdown_connections 2023-09-29T23:15:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:31.322+0000 7f4d95127700 1 -- 172.21.15.12:0/3730344495 wait complete. 2023-09-29T23:15:34.335 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:34.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.466+0000 7f243595f700 10 monclient: get_monmap_and_config 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.466+0000 7f243595f700 10 monclient: build_initial_monmap 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.466+0000 7f243595f700 10 monclient: monmap: 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:34.467206+0000 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:34.467206+0000 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:34.467 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.466+0000 7f243595f700 1 Processor -- start 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 -- start start 2023-09-29T23:15:34.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient: init 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient: _add_conns ranks=[5,10,9,0,8,1,6,4,2,7,3] 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 0x7f243009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): picked mon.noname-h con 0x7f24300a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f24300a2360 0x7f24300a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): picked mon.noname-f con 0x7f24300a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2430133740 0x7f2430133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): picked mon.noname-a con 0x7f2430133740 addr v2:172.21.15.12:3300/0 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243009e350 con 0x7f2430133740 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243009ef70 con 0x7f24300a2360 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f243009da70 con 0x7f24300a1f90 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f243595f700 10 monclient(hunting): authenticate will time out at 1243.827474s 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2430133740 0x7f2430133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f2430133740 0x7f2430133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58388/0 (socket says 172.21.15.12:58388) 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 -- 172.21.15.12:0/4158675018 learned_addr learned my addr 172.21.15.12:0/4158675018 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): get_auth_request con 0x7f2430133740 auth_method 0 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242e7fc700 1 --2- 172.21.15.12:0/4158675018 >> v2:172.21.15.153:3300/0 conn(0x7f24300a2360 0x7f24300a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242effd700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 0x7f243009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242e7fc700 10 monclient(hunting): get_auth_request con 0x7f24300a2360 auth_method 0 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient(hunting): handle_auth_done global_id 4653 payload 274 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 -- 172.21.15.12:0/4158675018 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 msgr2=0x7f243009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 0x7f243009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 -- 172.21.15.12:0/4158675018 >> v2:172.21.15.153:3300/0 conn(0x7f24300a2360 msgr2=0x7f24300a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 --2- 172.21.15.12:0/4158675018 >> v2:172.21.15.153:3300/0 conn(0x7f24300a2360 0x7f24300a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 monclient: found mon.noname-a 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:34.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242f7fe700 1 -- 172.21.15.12:0/4158675018 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2430068ed0 con 0x7f2430133740 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242effd700 10 monclient: get_auth_request con 0x7f24300a1f90 auth_method 0 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.467+0000 7f242effd700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 0x7f243009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242f7fe700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2430133740 0x7f2430133be0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24300a6ef0 tx=0x7f2428009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e38c8ceeb2cab8c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 1 -- 172.21.15.12:0/4158675018 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f242800bce0 con 0x7f2430133740 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: dump: 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.480 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:34.481 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 1 -- 172.21.15.12:0/4158675018 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242800be60 con 0x7f2430133740 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 1 -- 172.21.15.12:0/4158675018 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f24280118a0 con 0x7f2430133740 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f242dffb700 10 monclient: dump: 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.482 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:34.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 5 monclient: authenticate success, global_id 4653 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 4 set_mon_vals no callback set 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 10 monclient: shutdown 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 -- 172.21.15.12:0/4158675018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2430133740 msgr2=0x7f2430133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2430133740 0x7f2430133be0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24300a6ef0 tx=0x7f2428009250 comp rx=0 tx=0).stop 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 -- 172.21.15.12:0/4158675018 shutdown_connections 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2430133740 0x7f2430133be0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 --2- 172.21.15.12:0/4158675018 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f24300a1f90 0x7f243009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 --2- 172.21.15.12:0/4158675018 >> v2:172.21.15.153:3300/0 conn(0x7f24300a2360 0x7f24300a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 -- 172.21.15.12:0/4158675018 >> 172.21.15.12:0/4158675018 conn(0x7f24300a3b50 msgr2=0x7f24300a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.468+0000 7f243595f700 1 -- 172.21.15.12:0/4158675018 shutdown_connections 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.469+0000 7f243595f700 1 -- 172.21.15.12:0/4158675018 wait complete. 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.469+0000 7f243595f700 10 monclient: build_initial_monmap 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.469+0000 7f243595f700 10 monclient: monmap: 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:34.485 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.469+0000 7f243595f700 1 Processor -- start 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 -- start start 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient: init 2023-09-29T23:15:34.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient: _add_conns ranks=[2,7,5,1,0,6,3,8,4] 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): picked mon.noname-b con 0x7f24300a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f24300a2360 0x7f243009cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): picked mon.noname-i con 0x7f24300a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2430133740 0x7f243012d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): picked mon.noname-h con 0x7f2430133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24301309b0 con 0x7f24300a1f90 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f24300a0030 con 0x7f2430133740 2023-09-29T23:15:34.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2430133520 con 0x7f24300a2360 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f243595f700 10 monclient(hunting): authenticate will time out at 1243.830488s 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:58292/0 (socket says 172.21.15.12:58292) 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 -- 172.21.15.12:0/3529077020 learned_addr learned my addr 172.21.15.12:0/3529077020 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): get_auth_request con 0x7f24300a1f90 auth_method 0 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2430133740 0x7f243012d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f24300a2360 0x7f243009cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): get_auth_request con 0x7f2430133740 auth_method 0 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): get_auth_request con 0x7f24300a2360 auth_method 0 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:34.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 -- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f24300a2360 msgr2=0x7f243009cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f24300a2360 0x7f243009cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 -- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2430133740 msgr2=0x7f243012d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2430133740 0x7f243012d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient: _finish_hunting 0 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 monclient: found mon.noname-b 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.470+0000 7f242effd700 1 -- 172.21.15.12:0/3529077020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280094f0 con 0x7f24300a1f90 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f242effd700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f242000cfb0 tx=0x7f2420007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f242000ad80 con 0x7f24300a1f90 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: dump: 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24200168a0 con 0x7f24300a1f90 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 5 monclient: authenticate success, global_id 4475 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 10 monclient: _renew_subs 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243004c960 con 0x7f24300a1f90 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2420016e90 con 0x7f24300a1f90 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f2426ffd700 10 monclient: dump: 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f242d7fa700 4 set_mon_vals no callback set 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 10 monclient: _renew_subs 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.471+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243009e510 con 0x7f24300a1f90 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2420016a10 con 0x7f24300a1f90 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f243595f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:34.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f243595f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f243595f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2430070bc0 con 0x7f24300a1f90 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f2426ffd700 1 --2- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 0x7f2410038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2420012070 con 0x7f24300a1f90 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f242e7fc700 1 --2- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 0x7f2410038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.472+0000 7f242e7fc700 10 monclient: get_auth_request con 0x7f2410036780 auth_method 0 2023-09-29T23:15:34.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.473+0000 7f242e7fc700 1 --2- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 0x7f2410038c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2418006fd0 tx=0x7f2418008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.477+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2420016420 con 0x7f24300a1f90 2023-09-29T23:15:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.477+0000 7f2426ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:34.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.477+0000 7f2426ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.708+0000 7f243595f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.708+0000 7f243595f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.708+0000 7f243595f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.708+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2430130e20 con 0x7f24300a1f90 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.709+0000 7f2426ffd700 1 -- 172.21.15.12:0/3529077020 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2420014330 con 0x7f24300a1f90 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.709+0000 7f2426ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:34.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.709+0000 7f2426ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:34.710 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:34.710 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 msgr2=0x7f2410038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 0x7f2410038c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2418006fd0 tx=0x7f2418008040 comp rx=0 tx=0).stop 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 10 monclient: shutdown 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 msgr2=0x7f243009c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:34.711 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f242000cfb0 tx=0x7f2420007870 comp rx=0 tx=0).stop 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 shutdown_connections 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> v2:172.21.15.153:6812/83257 conn(0x7f2410036780 0x7f2410038c30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f24300a1f90 0x7f243009c700 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2430133740 0x7f243012d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 --2- 172.21.15.12:0/3529077020 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f24300a2360 0x7f243009cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 >> 172.21.15.12:0/3529077020 conn(0x7f24300a3b50 msgr2=0x7f243006e130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 shutdown_connections 2023-09-29T23:15:34.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:34.711+0000 7f243595f700 1 -- 172.21.15.12:0/3529077020 wait complete. 2023-09-29T23:15:37.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 10 monclient: get_monmap_and_config 2023-09-29T23:15:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 10 monclient: build_initial_monmap 2023-09-29T23:15:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 10 monclient: monmap: 2023-09-29T23:15:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:37.853 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:37.853924+0000 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:37.853924+0000 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:37.854 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:37.855 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:37.855 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:37.855 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:37.855 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:37.855 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 1 Processor -- start 2023-09-29T23:15:37.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 1 -- start start 2023-09-29T23:15:37.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.853+0000 7f2e86939700 10 monclient: init 2023-09-29T23:15:37.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient: _add_conns ranks=[10,5,3,7,4,1,9,0,8,2,6] 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 0x7f2e80131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-f con 0x7f2e80130fe0 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-h con 0x7f2e8006f0b0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 0x7f2e80096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-g con 0x7f2e800969e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e801313b0 con 0x7f2e80130fe0 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e80131e00 con 0x7f2e800969e0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8006d690 con 0x7f2e8006f0b0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e86939700 10 monclient(hunting): authenticate will time out at 1247.214397s 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 0x7f2e80131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 0x7f2e80096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:37596/0 (socket says 172.21.15.12:37596) 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 1 -- 172.21.15.12:0/4255702941 learned_addr learned my addr 172.21.15.12:0/4255702941 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): get_auth_request con 0x7f2e8006f0b0 auth_method 0 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.866 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): get_auth_request con 0x7f2e80130fe0 auth_method 0 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): get_auth_request con 0x7f2e800969e0 auth_method 0 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.854+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 -- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 msgr2=0x7f2e80096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 0x7f2e80096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 -- 172.21.15.12:0/4255702941 >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 msgr2=0x7f2e80131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 --2- 172.21.15.12:0/4255702941 >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 0x7f2e80131740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 monclient: found mon.noname-h 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 -- 172.21.15.12:0/4255702941 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80068ed0 con 0x7f2e8006f0b0 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7ffff700 1 --2- 172.21.15.12:0/4255702941 >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 0x7f2e80131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e84ed6700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 0x7f2e80096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7f7fe700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e680099b0 tx=0x7f2e68009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=600860860bd2237c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 1 -- 172.21.15.12:0/4255702941 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e6800bb00 con 0x7f2e8006f0b0 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:15:37.869 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: dump: 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:37.870 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: _finish_auth 0 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e86939700 5 monclient: authenticate success, global_id 4523 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 1 -- 172.21.15.12:0/4255702941 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800bc60 con 0x7f2e8006f0b0 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e86939700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:37.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e86939700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 1 -- 172.21.15.12:0/4255702941 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e6800bdd0 con 0x7f2e8006f0b0 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e86939700 4 set_mon_vals no callback set 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.855+0000 7f2e7effd700 10 monclient: dump: 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.872 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 10 monclient: shutdown 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 -- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 msgr2=0x7f2e8006f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e680099b0 tx=0x7f2e68009250 comp rx=0 tx=0).stop 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 -- 172.21.15.12:0/4255702941 shutdown_connections 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e800969e0 0x7f2e80096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 --2- 172.21.15.12:0/4255702941 >> v2:172.21.15.153:3300/0 conn(0x7f2e80130fe0 0x7f2e80131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 --2- 172.21.15.12:0/4255702941 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f2e8006f0b0 0x7f2e8006f500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 -- 172.21.15.12:0/4255702941 >> 172.21.15.12:0/4255702941 conn(0x7f2e800a4430 msgr2=0x7f2e800a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 -- 172.21.15.12:0/4255702941 shutdown_connections 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 1 -- 172.21.15.12:0/4255702941 wait complete. 2023-09-29T23:15:37.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 10 monclient: build_initial_monmap 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.856+0000 7f2e86939700 10 monclient: monmap: 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:37.875 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 Processor -- start 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 -- start start 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient: init 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient: _add_conns ranks=[3,1,0,6,5,7,2,4,8] 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8006f0b0 0x7f2e801421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-g con 0x7f2e8006f0b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e800969e0 0x7f2e8012f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-f con 0x7f2e800969e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): picked mon.noname-a con 0x7f2e80130fe0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e800a0180 con 0x7f2e80130fe0 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e80121f20 con 0x7f2e800969e0 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:37.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e80121c40 con 0x7f2e8006f0b0 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e86939700 10 monclient(hunting): authenticate will time out at 1247.217788s 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58400/0 (socket says 172.21.15.12:58400) 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 1 -- 172.21.15.12:0/2185428621 learned_addr learned my addr 172.21.15.12:0/2185428621 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 10 monclient(hunting): get_auth_request con 0x7f2e80130fe0 auth_method 0 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.857+0000 7f2e84ed6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e800969e0 0x7f2e8012f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8006f0b0 0x7f2e801421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): get_auth_request con 0x7f2e800969e0 auth_method 0 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): get_auth_request con 0x7f2e8006f0b0 auth_method 0 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:37.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient(hunting): handle_auth_done global_id 4662 payload 824 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 -- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8006f0b0 msgr2=0x7f2e801421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8006f0b0 0x7f2e801421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 -- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e800969e0 msgr2=0x7f2e8012f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e800969e0 0x7f2e8012f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient: _finish_hunting 0 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 monclient: found mon.noname-a 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 -- 172.21.15.12:0/2185428621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78009a20 con 0x7f2e80130fe0 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e84ed6700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009fc0 tx=0x7f2e7800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e7800cbd0 con 0x7f2e80130fe0 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: dump: 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: _finish_auth 0 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 5 monclient: authenticate success, global_id 4662 2023-09-29T23:15:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7800f040 con 0x7f2e80130fe0 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e78014d80 con 0x7f2e80130fe0 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e7e7fc700 4 set_mon_vals no callback set 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e77fff700 10 monclient: dump: 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:37.884 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:37.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 10 monclient: _renew_subs 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70009420 con 0x7f2e80130fe0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 10 monclient: _renew_subs 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.858+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e680094f0 con 0x7f2e80130fe0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2e78003020 con 0x7f2e80130fe0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e86939700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e86939700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e86939700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8004c960 con 0x7f2e80130fe0 2023-09-29T23:15:37.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e77fff700 1 --2- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 0x7f2e60038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2e78047b80 con 0x7f2e80130fe0 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.859+0000 7f2e7ffff700 1 --2- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 0x7f2e60038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.860+0000 7f2e7ffff700 10 monclient: get_auth_request con 0x7f2e60036850 auth_method 0 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.860+0000 7f2e7ffff700 1 --2- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 0x7f2e60038d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009eb0 tx=0x7f2e700091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.864+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e78012750 con 0x7f2e80130fe0 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.864+0000 7f2e77fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:37.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:37.864+0000 7f2e77fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.097+0000 7f2e86939700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.097+0000 7f2e86939700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.097+0000 7f2e86939700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.097+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2e80122090 con 0x7f2e80130fe0 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.097+0000 7f2e77fff700 1 -- 172.21.15.12:0/2185428621 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2e78018070 con 0x7f2e80130fe0 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.098+0000 7f2e77fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:38.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.098+0000 7f2e77fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:38.099 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:38.099 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 msgr2=0x7f2e60038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 0x7f2e60038d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009eb0 tx=0x7f2e700091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 10 monclient: shutdown 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 msgr2=0x7f2e8012fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009fc0 tx=0x7f2e7800b040 comp rx=0 tx=0).stop 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 shutdown_connections 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e80130fe0 0x7f2e8012fd30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> v2:172.21.15.153:6812/83257 conn(0x7f2e60036850 0x7f2e60038d00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:38.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8006f0b0 0x7f2e801421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:38.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 --2- 172.21.15.12:0/2185428621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e800969e0 0x7f2e8012f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:38.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.099+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 >> 172.21.15.12:0/2185428621 conn(0x7f2e800a4430 msgr2=0x7f2e800a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:38.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.100+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 shutdown_connections 2023-09-29T23:15:38.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:38.100+0000 7f2e86939700 1 -- 172.21.15.12:0/2185428621 wait complete. 2023-09-29T23:15:41.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 10 monclient: get_monmap_and_config 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 10 monclient: build_initial_monmap 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 10 monclient: monmap: 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:41.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:41.244163+0000 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:41.244163+0000 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:41.244 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:41.245 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:41.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 1 Processor -- start 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 1 -- start start 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.243+0000 7f60802be700 10 monclient: init 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient: _add_conns ranks=[10,3,4,2,5,9,6,7,1,0,8] 2023-09-29T23:15:41.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f60780a1f90 0x7f607809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): picked mon.noname-f con 0x7f60780a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60780a2360 0x7f60780a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): picked mon.noname-g con 0x7f60780a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): picked mon.noname-c con 0x7f6078133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607809e350 con 0x7f60780a1f90 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607809ef70 con 0x7f60780a2360 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f607809da70 con 0x7f6078133740 2023-09-29T23:15:41.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f60802be700 10 monclient(hunting): authenticate will time out at 1250.604635s 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:39170/0 (socket says 172.21.15.12:39170) 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 1 -- 172.21.15.12:0/133387793 learned_addr learned my addr 172.21.15.12:0/133387793 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 10 monclient(hunting): get_auth_request con 0x7f6078133740 auth_method 0 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607d859700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60780a2360 0x7f60780a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607d859700 10 monclient(hunting): get_auth_request con 0x7f60780a2360 auth_method 0 2023-09-29T23:15:41.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607d859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e05a700 1 --2- 172.21.15.12:0/133387793 >> v2:172.21.15.153:3300/0 conn(0x7f60780a1f90 0x7f607809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607d859700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607d859700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e05a700 10 monclient(hunting): get_auth_request con 0x7f60780a1f90 auth_method 0 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e05a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e05a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e05a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.244+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 10 monclient(hunting): handle_auth_done global_id 4567 payload 274 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 -- 172.21.15.12:0/133387793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60780a2360 msgr2=0x7f60780a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:41.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60780a2360 0x7f60780a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 -- 172.21.15.12:0/133387793 >> v2:172.21.15.153:3300/0 conn(0x7f60780a1f90 msgr2=0x7f607809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 --2- 172.21.15.12:0/133387793 >> v2:172.21.15.153:3300/0 conn(0x7f60780a1f90 0x7f607809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 10 monclient: _finish_hunting 0 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 monclient: found mon.noname-c 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 -- 172.21.15.12:0/133387793 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078068ed0 con 0x7f6078133740 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607e85b700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6070009a10 tx=0x7f6070009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=20772bf4ddf176c6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 1 -- 172.21.15.12:0/133387793 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6070011ab0 con 0x7f6078133740 2023-09-29T23:15:41.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: dump: 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:41.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: _finish_auth 0 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:41.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 1 -- 172.21.15.12:0/133387793 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6070011c30 con 0x7f6078133740 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f60802be700 5 monclient: authenticate success, global_id 4567 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f60802be700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f60802be700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 1 -- 172.21.15.12:0/133387793 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6070011da0 con 0x7f6078133740 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f60802be700 4 set_mon_vals no callback set 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.245+0000 7f607d058700 10 monclient: dump: 2023-09-29T23:15:41.263 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 10 monclient: shutdown 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 -- 172.21.15.12:0/133387793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 msgr2=0x7f6078133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6070009a10 tx=0x7f6070009250 comp rx=0 tx=0).stop 2023-09-29T23:15:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 -- 172.21.15.12:0/133387793 shutdown_connections 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f6078133740 0x7f6078133be0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 --2- 172.21.15.12:0/133387793 >> v2:172.21.15.153:3300/0 conn(0x7f60780a1f90 0x7f607809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 --2- 172.21.15.12:0/133387793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60780a2360 0x7f60780a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 -- 172.21.15.12:0/133387793 >> 172.21.15.12:0/133387793 conn(0x7f60780a3b50 msgr2=0x7f60780a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 -- 172.21.15.12:0/133387793 shutdown_connections 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 1 -- 172.21.15.12:0/133387793 wait complete. 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 10 monclient: build_initial_monmap 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.246+0000 7f60802be700 10 monclient: monmap: 2023-09-29T23:15:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:41.267 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 Processor -- start 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 -- start start 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient: init 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient: _add_conns ranks=[6,5,2,3,1,7,8,4,0] 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): picked mon.noname-d con 0x7f60780a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f60780a2360 0x7f60781328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): picked mon.noname-h con 0x7f60780a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6078133740 0x7f607812d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): picked mon.noname-b con 0x7f6078133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6078142f20 con 0x7f6078133740 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f607812f0e0 con 0x7f60780a2360 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6078130df0 con 0x7f60780a1f90 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:41.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f60802be700 10 monclient(hunting): authenticate will time out at 1250.607614s 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6078133740 0x7f607812d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:33584/0 (socket says 172.21.15.12:33584) 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 1 -- 172.21.15.12:0/962822247 learned_addr learned my addr 172.21.15.12:0/962822247 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): get_auth_request con 0x7f60780a1f90 auth_method 0 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): get_auth_request con 0x7f6078133740 auth_method 0 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f60780a2360 0x7f60781328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e05a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): get_auth_request con 0x7f60780a2360 auth_method 0 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607e85b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.247+0000 7f607d859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 -- 172.21.15.12:0/962822247 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f60780a2360 msgr2=0x7f60781328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f60780a2360 0x7f60781328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 -- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6078133740 msgr2=0x7f607812d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6078133740 0x7f607812d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 10 monclient: _finish_hunting 0 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 monclient: found mon.noname-d 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:41.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 -- 172.21.15.12:0/962822247 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60700094f0 con 0x7f60780a1f90 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607e05a700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6074009eb0 tx=0x7f607400b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6074014480 con 0x7f60780a1f90 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: dump: 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:41.273 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:15:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607400f040 con 0x7f60780a1f90 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 5 monclient: authenticate success, global_id 4605 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 10 monclient: _renew_subs 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607804c960 con 0x7f60780a1f90 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6074014ac0 con 0x7f60780a1f90 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f606e7fc700 10 monclient: dump: 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f607c857700 4 set_mon_vals no callback set 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 10 monclient: _renew_subs 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.248+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781310d0 con 0x7f60780a1f90 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6074003020 con 0x7f60780a1f90 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f60802be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f60802be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f60802be700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6078070bc0 con 0x7f60780a1f90 2023-09-29T23:15:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f606e7fc700 1 --2- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 0x7f605c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f60740478d0 con 0x7f60780a1f90 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f607d859700 1 --2- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 0x7f605c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.249+0000 7f607d859700 10 monclient: get_auth_request con 0x7f605c036880 auth_method 0 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.250+0000 7f607d859700 1 --2- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 0x7f605c038d30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f607812d390 tx=0x7f60680091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.254+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f607401a2c0 con 0x7f60780a1f90 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.254+0000 7f606e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.254+0000 7f606e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.493+0000 7f60802be700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.493+0000 7f60802be700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.493+0000 7f60802be700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.493+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6078066790 con 0x7f60780a1f90 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.494+0000 7f606e7fc700 1 -- 172.21.15.12:0/962822247 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6074018070 con 0x7f60780a1f90 2023-09-29T23:15:41.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.494+0000 7f606e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:41.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.494+0000 7f606e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:41.495 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:41.495 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:41.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 msgr2=0x7f605c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 0x7f605c038d30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f607812d390 tx=0x7f60680091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 10 monclient: shutdown 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 msgr2=0x7f6078142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6074009eb0 tx=0x7f607400b040 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 shutdown_connections 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> v2:172.21.15.153:6812/83257 conn(0x7f605c036880 0x7f605c038d30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6078133740 0x7f607812d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f60780a1f90 0x7f6078142020 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 --2- 172.21.15.12:0/962822247 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f60780a2360 0x7f60781328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 >> 172.21.15.12:0/962822247 conn(0x7f60780a3b50 msgr2=0x7f6078135bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:41.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 shutdown_connections 2023-09-29T23:15:41.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:41.496+0000 7f60802be700 1 -- 172.21.15.12:0/962822247 wait complete. 2023-09-29T23:15:44.508 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:44.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.638+0000 7f80c14ef700 10 monclient: get_monmap_and_config 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.638+0000 7f80c14ef700 10 monclient: build_initial_monmap 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.638+0000 7f80c14ef700 10 monclient: monmap: 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:44.639744+0000 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:44.639744+0000 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:44.639 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:44.640 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 Processor -- start 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 -- start start 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient: init 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient: _add_conns ranks=[4,6,3,1,10,0,8,5,7,2,9] 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-c con 0x7f80bc0a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 0x7f80bc0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-d con 0x7f80bc0a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80bc133740 0x7f80bc133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-g con 0x7f80bc133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc09e350 con 0x7f80bc133740 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc09ef70 con 0x7f80bc0a1f90 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc09da70 con 0x7f80bc0a2360 2023-09-29T23:15:44.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.639+0000 7f80c14ef700 10 monclient(hunting): authenticate will time out at 1253.999998s 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 0x7f80bc0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44774/0 (socket says 172.21.15.12:44774) 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 -- 172.21.15.12:0/3248481316 learned_addr learned my addr 172.21.15.12:0/3248481316 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): get_auth_request con 0x7f80bc0a1f90 auth_method 0 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): get_auth_request con 0x7f80bc0a2360 auth_method 0 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80bb7fe700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80bc133740 0x7f80bc133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80bb7fe700 10 monclient(hunting): get_auth_request con 0x7f80bc133740 auth_method 0 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient(hunting): handle_auth_done global_id 4576 payload 274 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 -- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 msgr2=0x7f80bc0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 0x7f80bc0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 -- 172.21.15.12:0/3248481316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80bc133740 msgr2=0x7f80bc133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80bc133740 0x7f80bc133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient: _finish_hunting 0 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 monclient: found mon.noname-c 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 -- 172.21.15.12:0/3248481316 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc068ed0 con 0x7f80bc0a1f90 2023-09-29T23:15:44.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80ba7fc700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 0x7f80bc0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80baffd700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009a10 tx=0x7f80b4009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=75b05e3b453570aa server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 1 -- 172.21.15.12:0/3248481316 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80b400bca0 con 0x7f80bc0a1f90 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: dump: 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:44.654 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: _finish_auth 0 2023-09-29T23:15:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 1 -- 172.21.15.12:0/3248481316 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b400be20 con 0x7f80bc0a1f90 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80c14ef700 5 monclient: authenticate success, global_id 4576 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80c14ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80c14ef700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 1 -- 172.21.15.12:0/3248481316 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80b40118a0 con 0x7f80bc0a1f90 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80c14ef700 4 set_mon_vals no callback set 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.640+0000 7f80b9ffb700 10 monclient: dump: 2023-09-29T23:15:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 10 monclient: shutdown 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 -- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 msgr2=0x7f80bc09f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009a10 tx=0x7f80b4009250 comp rx=0 tx=0).stop 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 -- 172.21.15.12:0/3248481316 shutdown_connections 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80bc133740 0x7f80bc133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80bc0a1f90 0x7f80bc09f5e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3248481316 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80bc0a2360 0x7f80bc0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 -- 172.21.15.12:0/3248481316 >> 172.21.15.12:0/3248481316 conn(0x7f80bc0a3b50 msgr2=0x7f80bc0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 -- 172.21.15.12:0/3248481316 shutdown_connections 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.641+0000 7f80c14ef700 1 -- 172.21.15.12:0/3248481316 wait complete. 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.642+0000 7f80c14ef700 10 monclient: build_initial_monmap 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.642+0000 7f80c14ef700 10 monclient: monmap: 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:44.659 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.642+0000 7f80c14ef700 1 Processor -- start 2023-09-29T23:15:44.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.642+0000 7f80c14ef700 1 -- start start 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.642+0000 7f80c14ef700 10 monclient: init 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient: _add_conns ranks=[7,5,2,1,8,6,0,4,3] 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 0x7f80bc131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-i con 0x7f80bc0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f80bc0a2360 0x7f80bc132b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-h con 0x7f80bc0a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): picked mon.noname-b con 0x7f80bc133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc131030 con 0x7f80bc133740 2023-09-29T23:15:44.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc131310 con 0x7f80bc0a2360 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc130d50 con 0x7f80bc0a1f90 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient(hunting): authenticate will time out at 1254.003192s 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 0x7f80bc131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 0x7f80bc131fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:55314/0 (socket says 172.21.15.12:55314) 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54454/0 (socket says 172.21.15.12:54454) 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 1 -- 172.21.15.12:0/3002835531 learned_addr learned my addr 172.21.15.12:0/3002835531 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:44.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): get_auth_request con 0x7f80bc0a1f90 auth_method 0 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): get_auth_request con 0x7f80bc133740 auth_method 0 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f80bc0a2360 0x7f80bc132b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): get_auth_request con 0x7f80bc0a2360 auth_method 0 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:44.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2023-09-29T23:15:44.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 -- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 msgr2=0x7f80bc131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 0x7f80bc131fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 -- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f80bc0a2360 msgr2=0x7f80bc132b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f80bc0a2360 0x7f80bc132b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 monclient: found mon.noname-b 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 -- 172.21.15.12:0/3002835531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4009420 con 0x7f80bc133740 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80bb7fe700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80ac007e40 tx=0x7f80ac0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80ac009840 con 0x7f80bc133740 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 10 monclient: dump: 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 10 monclient: _finish_auth 0 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac014420 con 0x7f80bc133740 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 5 monclient: authenticate success, global_id 4493 2023-09-29T23:15:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient: _renew_subs 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b40094f0 con 0x7f80bc133740 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.643+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80ac014a10 con 0x7f80bc133740 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80b2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80b2ffd700 10 monclient: dump: 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:44.669 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80b97fa700 4 set_mon_vals no callback set 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 10 monclient: _renew_subs 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc04c960 con 0x7f80bc133740 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f80ac003020 con 0x7f80bc133740 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.644+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc09e490 con 0x7f80bc133740 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.645+0000 7f80b2ffd700 1 --2- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 0x7f809c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.645+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f80ac047570 con 0x7f80bc133740 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.645+0000 7f80baffd700 1 --2- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 0x7f809c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.645+0000 7f80baffd700 10 monclient: get_auth_request con 0x7f809c036850 auth_method 0 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.645+0000 7f80baffd700 1 --2- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 0x7f809c038d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f80b40094c0 tx=0x7f80b4017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.649+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80ac014590 con 0x7f80bc133740 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.650+0000 7f80b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.650+0000 7f80b2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80c14ef700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:44.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80c14ef700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80c14ef700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f80bc070bc0 con 0x7f80bc133740 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80b2ffd700 1 -- 172.21.15.12:0/3002835531 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f80ac01dbb0 con 0x7f80bc133740 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80b2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.885+0000 7f80b2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:44.886 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 msgr2=0x7f809c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 0x7f809c038d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f80b40094c0 tx=0x7f80b4017000 comp rx=0 tx=0).stop 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 10 monclient: shutdown 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 msgr2=0x7f80bc12d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80ac007e40 tx=0x7f80ac0097b0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 shutdown_connections 2023-09-29T23:15:44.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> v2:172.21.15.153:6812/83257 conn(0x7f809c036850 0x7f809c038d00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f80bc133740 0x7f80bc12d5d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f80bc0a1f90 0x7f80bc131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 --2- 172.21.15.12:0/3002835531 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f80bc0a2360 0x7f80bc132b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.887+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 >> 172.21.15.12:0/3002835531 conn(0x7f80bc0a3b50 msgr2=0x7f80bc06f280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.888+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 shutdown_connections 2023-09-29T23:15:44.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:44.888+0000 7f80c14ef700 1 -- 172.21.15.12:0/3002835531 wait complete. 2023-09-29T23:15:47.900 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:48.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.029+0000 7fe4cf1cc700 10 monclient: get_monmap_and_config 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.029+0000 7fe4cf1cc700 10 monclient: build_initial_monmap 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.029+0000 7fe4cf1cc700 10 monclient: monmap: 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:48.030617+0000 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:48.030617+0000 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:48.030 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:48.031 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.030+0000 7fe4cf1cc700 1 Processor -- start 2023-09-29T23:15:48.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.030+0000 7fe4cf1cc700 1 -- start start 2023-09-29T23:15:48.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.030+0000 7fe4cf1cc700 10 monclient: init 2023-09-29T23:15:48.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:48.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient: _add_conns ranks=[4,10,2,3,8,9,1,0,5,7,6] 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-c con 0x7fe4c80a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe4c80a2360 0x7fe4c80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-f con 0x7fe4c80a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-b con 0x7fe4c8133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c809e350 con 0x7fe4c80a2360 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c809ef70 con 0x7fe4c8133740 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c809da70 con 0x7fe4c80a1f90 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cf1cc700 10 monclient(hunting): authenticate will time out at 1257.391181s 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54462/0 (socket says 172.21.15.12:54462) 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44790/0 (socket says 172.21.15.12:44790) 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 1 -- 172.21.15.12:0/2788382849 learned_addr learned my addr 172.21.15.12:0/2788382849 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): get_auth_request con 0x7fe4c8133740 auth_method 0 2023-09-29T23:15:48.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): get_auth_request con 0x7fe4c80a1f90 auth_method 0 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 1 --2- 172.21.15.12:0/2788382849 >> v2:172.21.15.153:3300/0 conn(0x7fe4c80a2360 0x7fe4c80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): get_auth_request con 0x7fe4c80a2360 auth_method 0 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:48.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient(hunting): handle_auth_done global_id 4585 payload 274 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 -- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 msgr2=0x7fe4c8133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 -- 172.21.15.12:0/2788382849 >> v2:172.21.15.153:3300/0 conn(0x7fe4c80a2360 msgr2=0x7fe4c80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/2788382849 >> v2:172.21.15.153:3300/0 conn(0x7fe4c80a2360 0x7fe4c80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient: _finish_hunting 0 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 monclient: found mon.noname-c 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 -- 172.21.15.12:0/2788382849 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8068ed0 con 0x7fe4c80a1f90 2023-09-29T23:15:48.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4cd769700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.031+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009940 tx=0x7fe4b00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7c8b095025f6aac4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 1 -- 172.21.15.12:0/2788382849 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4b0011ab0 con 0x7fe4c80a1f90 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: dump: 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:48.045 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: _finish_auth 0 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 1 -- 172.21.15.12:0/2788382849 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0011c30 con 0x7fe4c80a1f90 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 5 monclient: authenticate success, global_id 4585 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 1 -- 172.21.15.12:0/2788382849 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4b0011da0 con 0x7fe4c80a1f90 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 4 set_mon_vals no callback set 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4c77fe700 10 monclient: dump: 2023-09-29T23:15:48.047 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:48.048 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 10 monclient: shutdown 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 msgr2=0x7fe4c809f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009940 tx=0x7fe4b00091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/2788382849 shutdown_connections 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe4c8133740 0x7fe4c8133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/2788382849 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c80a1f90 0x7fe4c809f5e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/2788382849 >> v2:172.21.15.153:3300/0 conn(0x7fe4c80a2360 0x7fe4c80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/2788382849 >> 172.21.15.12:0/2788382849 conn(0x7fe4c80a3b50 msgr2=0x7fe4c80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/2788382849 shutdown_connections 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.032+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/2788382849 wait complete. 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.033+0000 7fe4cf1cc700 10 monclient: build_initial_monmap 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.033+0000 7fe4cf1cc700 10 monclient: monmap: 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:48.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.033+0000 7fe4cf1cc700 1 Processor -- start 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.033+0000 7fe4cf1cc700 1 -- start start 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.033+0000 7fe4cf1cc700 10 monclient: init 2023-09-29T23:15:48.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient: _add_conns ranks=[3,8,4,2,7,1,5,0,6] 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 0x7fe4c81420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-g con 0x7fe4c80a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 0x7fe4c81328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-e con 0x7fe4c80a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): picked mon.noname-c con 0x7fe4c8133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c809e510 con 0x7fe4c80a1f90 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8130de0 con 0x7fe4c8133740 2023-09-29T23:15:48.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8130b00 con 0x7fe4c80a2360 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cf1cc700 10 monclient(hunting): authenticate will time out at 1257.394329s 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 0x7fe4c81328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44806/0 (socket says 172.21.15.12:44806) 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 -- 172.21.15.12:0/1621207821 learned_addr learned my addr 172.21.15.12:0/1621207821 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): get_auth_request con 0x7fe4c8133740 auth_method 0 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): get_auth_request con 0x7fe4c80a2360 auth_method 0 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:48.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 0x7fe4c81420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 10 monclient(hunting): get_auth_request con 0x7fe4c80a1f90 auth_method 0 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 -- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 msgr2=0x7fe4c81328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 0x7fe4c81328d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 -- 172.21.15.12:0/1621207821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 msgr2=0x7fe4c81420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 0x7fe4c81420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient: _finish_hunting 0 2023-09-29T23:15:48.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 monclient: found mon.noname-c 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 -- 172.21.15.12:0/1621207821 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c00094f0 con 0x7fe4c8133740 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 0x7fe4c81420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4c7fff700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 0x7fe4c81328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.034+0000 7fe4cd769700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000ec50 tx=0x7fe4c0009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4c000ccd0 con 0x7fe4c8133740 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: dump: 2023-09-29T23:15:48.056 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:48.057 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: _finish_auth 0 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 5 monclient: authenticate success, global_id 4594 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c000ce50 con 0x7fe4c8133740 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 10 monclient: _renew_subs 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b0009420 con 0x7fe4c8133740 2023-09-29T23:15:48.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4c00138f0 con 0x7fe4c8133740 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c6ffd700 4 set_mon_vals no callback set 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4c4ff9700 10 monclient: dump: 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:48.059 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:48.060 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 10 monclient: _renew_subs 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.035+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c804c960 con 0x7fe4c8133740 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe4c0013470 con 0x7fe4c8133740 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4cf1cc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4cf1cc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4cf1cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c8070bc0 con 0x7fe4c8133740 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4c4ff9700 1 --2- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 0x7fe4a8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe4c0047520 con 0x7fe4c8133740 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 0x7fe4a8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.036+0000 7fe4ccf68700 10 monclient: get_auth_request con 0x7fe4a8036850 auth_method 0 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.037+0000 7fe4ccf68700 1 --2- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 0x7fe4a8038d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009940 tx=0x7fe4b0017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:48.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.041+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4c0022d00 con 0x7fe4c8133740 2023-09-29T23:15:48.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.041+0000 7fe4c4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:48.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.041+0000 7fe4c4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4cf1cc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4cf1cc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4cf1cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4c812f030 con 0x7fe4c8133740 2023-09-29T23:15:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4c4ff9700 1 -- 172.21.15.12:0/1621207821 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe4c00222f0 con 0x7fe4c8133740 2023-09-29T23:15:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4c4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:48.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.273+0000 7fe4c4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:48.274 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 msgr2=0x7fe4a8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 0x7fe4a8038d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009940 tx=0x7fe4b0017000 comp rx=0 tx=0).stop 2023-09-29T23:15:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 10 monclient: shutdown 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 msgr2=0x7fe4c812d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000ec50 tx=0x7fe4c0009250 comp rx=0 tx=0).stop 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 shutdown_connections 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe4c8133740 0x7fe4c812d350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> v2:172.21.15.153:6812/83257 conn(0x7fe4a8036850 0x7fe4a8038d00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4c80a1f90 0x7fe4c81420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 --2- 172.21.15.12:0/1621207821 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe4c80a2360 0x7fe4c81328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 >> 172.21.15.12:0/1621207821 conn(0x7fe4c80a3b50 msgr2=0x7fe4c8135c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 shutdown_connections 2023-09-29T23:15:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:48.275+0000 7fe4cf1cc700 1 -- 172.21.15.12:0/1621207821 wait complete. 2023-09-29T23:15:51.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:51.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.432+0000 7f455c609700 10 monclient: get_monmap_and_config 2023-09-29T23:15:51.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.432+0000 7f455c609700 10 monclient: build_initial_monmap 2023-09-29T23:15:51.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.432+0000 7f455c609700 10 monclient: monmap: 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:51.433320+0000 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:51.433320+0000 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:51.433 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:51.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 Processor -- start 2023-09-29T23:15:51.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 -- start start 2023-09-29T23:15:51.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient: init 2023-09-29T23:15:51.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:51.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient: _add_conns ranks=[3,7,0,4,8,6,2,10,1,9,5] 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 0x7f455409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): picked mon.noname-g con 0x7f45540a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 0x7f45540a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): picked mon.noname-i con 0x7f45540a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4554133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:51.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4554068ed0 con 0x7f4554133740 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455409e350 con 0x7f45540a1f90 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f455409ef70 con 0x7f45540a2360 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455c609700 10 monclient(hunting): authenticate will time out at 1260.793800s 2023-09-29T23:15:51.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455aba6700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f4554133740 0x7f4554133bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:58576/0 (socket says 172.21.15.12:58576) 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.433+0000 7f455aba6700 1 -- 172.21.15.12:0/3927557376 learned_addr learned my addr 172.21.15.12:0/3927557376 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 0x7f455409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 0x7f45540a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): get_auth_request con 0x7f45540a1f90 auth_method 0 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): get_auth_request con 0x7f45540a2360 auth_method 0 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113321201 0 0) 0x7f4554068ed0 con 0x7f4554133740 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:51.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient(hunting): my global_id is 4671 2023-09-29T23:15:51.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c003570 con 0x7f4554133740 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient(hunting): handle_auth_done global_id 4503 payload 274 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 -- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 msgr2=0x7f45540a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218811956 0 0) 0x7f453c003570 con 0x7f4554133740 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 0x7f45540a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 -- 172.21.15.12:0/3927557376 >> v1:172.21.15.12:6789/0 conn(0x7f4554133740 legacy=0x7f4554133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient: _finish_hunting 0 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 monclient: found mon.noname-g 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.153:3301/0 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 -- 172.21.15.12:0/3927557376 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455404c960 con 0x7f45540a1f90 2023-09-29T23:15:51.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f4559ba4700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 0x7f45540a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455a3a5700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 0x7f455409f5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4544009ec0 tx=0x7f4544009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=d924d44c9d4f485 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 <== mon.3 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f454400e070 con 0x7f45540a1f90 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: dump: 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.451 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:51.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: _finish_auth 0 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 <== mon.3 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45440113b0 con 0x7f45540a1f90 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455c609700 5 monclient: authenticate success, global_id 4503 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455c609700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455c609700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 1 -- 172.21.15.12:0/3927557376 <== mon.3 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f454c0018f0 con 0x7f45540a1f90 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f45593a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.434+0000 7f455c609700 4 set_mon_vals no callback set 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f45593a3700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-29T23:15:51.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f45593a3700 10 monclient: dump: 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:51.454 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 10 monclient: shutdown 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 -- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 msgr2=0x7f455409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 0x7f455409f5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4544009ec0 tx=0x7f4544009780 comp rx=0 tx=0).stop 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 -- 172.21.15.12:0/3927557376 shutdown_connections 2023-09-29T23:15:51.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45540a1f90 0x7f455409f5e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 --2- 172.21.15.12:0/3927557376 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f45540a2360 0x7f45540a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 -- 172.21.15.12:0/3927557376 >> 172.21.15.12:0/3927557376 conn(0x7f45540a3b50 msgr2=0x7f45540a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 -- 172.21.15.12:0/3927557376 shutdown_connections 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.435+0000 7f455c609700 1 -- 172.21.15.12:0/3927557376 wait complete. 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient: build_initial_monmap 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient: monmap: 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:51.456 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 Processor -- start 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 -- start start 2023-09-29T23:15:51.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient: init 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient: _add_conns ranks=[8,5,3,2,1,4,7,6,0] 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): picked mon.noname-e con 0x7f45540a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f45540a2360 0x7f455409cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): picked mon.noname-h con 0x7f45540a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 0x7f455412d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): picked mon.noname-g con 0x7f4554133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455409ffc0 con 0x7f4554133740 2023-09-29T23:15:51.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f455412fae0 con 0x7f45540a2360 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f45541309d0 con 0x7f45540a1f90 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.436+0000 7f455c609700 10 monclient(hunting): authenticate will time out at 1260.796935s 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56326/0 (socket says 172.21.15.12:56326) 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 -- 172.21.15.12:0/1046075115 learned_addr learned my addr 172.21.15.12:0/1046075115 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): get_auth_request con 0x7f45540a1f90 auth_method 0 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f45540a2360 0x7f455409cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 0x7f455412d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): get_auth_request con 0x7f45540a2360 auth_method 0 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 10 monclient(hunting): get_auth_request con 0x7f4554133740 auth_method 0 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4559ba4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 -- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f45540a2360 msgr2=0x7f455409cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f45540a2360 0x7f455409cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 -- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 msgr2=0x7f455412d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 0x7f455412d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient: _finish_hunting 0 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 monclient: found mon.noname-e 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 -- 172.21.15.12:0/1046075115 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548009420 con 0x7f45540a1f90 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455aba6700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 0x7f455412d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455a3a5700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4544003000 tx=0x7f4544017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4544009780 con 0x7f45540a1f90 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: dump: 2023-09-29T23:15:51.462 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:51.463 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: _finish_auth 0 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 5 monclient: authenticate success, global_id 4481 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 10 monclient: _renew_subs 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45440128d0 con 0x7f45540a1f90 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4544009a20 con 0x7f45540a1f90 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:51.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f454401e430 con 0x7f45540a1f90 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f4558ba2700 4 set_mon_vals no callback set 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f45527fc700 10 monclient: dump: 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:51.465 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 10 monclient: _renew_subs 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.437+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455412eea0 con 0x7f45540a1f90 2023-09-29T23:15:51.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4544012450 con 0x7f45540a1f90 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f455c609700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f455c609700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f455c609700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45541310e0 con 0x7f45540a1f90 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f45527fc700 1 --2- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 0x7f4540038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.438+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4544048910 con 0x7f45540a1f90 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.439+0000 7f4559ba4700 1 --2- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 0x7f4540038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.439+0000 7f4559ba4700 10 monclient: get_auth_request con 0x7f4540036850 auth_method 0 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.439+0000 7f4559ba4700 1 --2- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 0x7f4540038d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f455412d250 tx=0x7f45480091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.443+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f454401cb60 con 0x7f45540a1f90 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.443+0000 7f45527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:51.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.443+0000 7f45527fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f455c609700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f455c609700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f455c609700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4554130b10 con 0x7f45540a1f90 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f45527fc700 1 -- 172.21.15.12:0/1046075115 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4554130b10 con 0x7f45540a1f90 2023-09-29T23:15:51.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f45527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:51.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.682+0000 7f45527fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:51.683 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:51.683 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:51.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 msgr2=0x7f4540038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 0x7f4540038d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f455412d250 tx=0x7f45480091a0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 10 monclient: shutdown 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 msgr2=0x7f455409c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4544003000 tx=0x7f4544017000 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 shutdown_connections 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> v2:172.21.15.153:6812/83257 conn(0x7f4540036850 0x7f4540038d00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4554133740 0x7f455412d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f45540a1f90 0x7f455409c700 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 --2- 172.21.15.12:0/1046075115 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f45540a2360 0x7f455409cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:51.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 >> 172.21.15.12:0/1046075115 conn(0x7f45540a3b50 msgr2=0x7f45540a6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 shutdown_connections 2023-09-29T23:15:51.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:51.684+0000 7f455c609700 1 -- 172.21.15.12:0/1046075115 wait complete. 2023-09-29T23:15:54.699 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.824+0000 7fdb88897700 10 monclient: get_monmap_and_config 2023-09-29T23:15:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.824+0000 7fdb88897700 10 monclient: build_initial_monmap 2023-09-29T23:15:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.824+0000 7fdb88897700 10 monclient: monmap: 2023-09-29T23:15:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:54.825 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:54.825626+0000 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:54.825626+0000 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:54.826 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.825+0000 7fdb88897700 1 Processor -- start 2023-09-29T23:15:54.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.825+0000 7fdb88897700 1 -- start start 2023-09-29T23:15:54.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.825+0000 7fdb88897700 10 monclient: init 2023-09-29T23:15:54.837 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.825+0000 7fdb88897700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.825+0000 7fdb88897700 10 monclient: _add_conns ranks=[0,4,5,2,10,1,8,6,7,3,9] 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb840a1f90 addr v1:172.21.15.12:6789/0 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-c con 0x7fdb840a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb84138980 0x7fdb8413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-h con 0x7fdb84138980 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb84068ed0 con 0x7fdb840a1f90 2023-09-29T23:15:54.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8409e350 con 0x7fdb840a2360 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8409fcf0 con 0x7fdb84138980 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient(hunting): authenticate will time out at 1264.186077s 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:40188/0 (socket says 172.21.15.12:40188) 2023-09-29T23:15:54.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8259c700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fdb840a1f90 0x7fdb8409f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39040/0 (socket says 172.21.15.12:39040) 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 -- 172.21.15.12:0/1721030914 learned_addr learned my addr 172.21.15.12:0/1721030914 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): get_auth_request con 0x7fdb840a2360 auth_method 0 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb84138980 0x7fdb8413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): get_auth_request con 0x7fdb84138980 auth_method 0 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 1 -- 172.21.15.12:0/1721030914 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721588983 0 0) 0x7fdb84068ed0 con 0x7fdb840a1f90 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient(hunting): handle_auth_done global_id 4603 payload 274 2023-09-29T23:15:54.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 -- 172.21.15.12:0/1721030914 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb84138980 msgr2=0x7fdb8413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb84138980 0x7fdb8413ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 -- 172.21.15.12:0/1721030914 >> v1:172.21.15.12:6789/0 conn(0x7fdb840a1f90 legacy=0x7fdb8409f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 monclient: found mon.noname-c 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 -- 172.21.15.12:0/1721030914 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8404c960 con 0x7fdb840a2360 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:15:54.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb81d9b700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdb780099e0 tx=0x7fdb78009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1087338cf676605d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 1 -- 172.21.15.12:0/1721030914 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb7800bc80 con 0x7fdb840a2360 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: dump: 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:54.843 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:54.844 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: _finish_auth 0 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 5 monclient: authenticate success, global_id 4603 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 1 -- 172.21.15.12:0/1721030914 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7800be00 con 0x7fdb840a2360 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb88897700 4 set_mon_vals no callback set 2023-09-29T23:15:54.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 1 -- 172.21.15.12:0/1721030914 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb780118a0 con 0x7fdb840a2360 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.826+0000 7fdb8159a700 10 monclient: dump: 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:54.846 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:54.847 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 10 monclient: shutdown 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 -- 172.21.15.12:0/1721030914 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 msgr2=0x7fdb840a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdb780099e0 tx=0x7fdb78009250 comp rx=0 tx=0).stop 2023-09-29T23:15:54.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 -- 172.21.15.12:0/1721030914 shutdown_connections 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb84138980 0x7fdb8413ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 --2- 172.21.15.12:0/1721030914 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb840a2360 0x7fdb840a27b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 -- 172.21.15.12:0/1721030914 >> 172.21.15.12:0/1721030914 conn(0x7fdb840a3b50 msgr2=0x7fdb840a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 -- 172.21.15.12:0/1721030914 shutdown_connections 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.827+0000 7fdb88897700 1 -- 172.21.15.12:0/1721030914 wait complete. 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient: build_initial_monmap 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient: monmap: 2023-09-29T23:15:54.849 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:54.850 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:54.851 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:54.852 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 Processor -- start 2023-09-29T23:15:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 -- start start 2023-09-29T23:15:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient: init 2023-09-29T23:15:54.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient: _add_conns ranks=[5,1,4,6,2,7,8,0,3] 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 0x7fdb84142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-h con 0x7fdb840a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb840a2360 0x7fdb84097900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-f con 0x7fdb840a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): picked mon.noname-c con 0x7fdb84138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8408fdb0 con 0x7fdb840a2360 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdb84143020 con 0x7fdb84138980 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:54.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8409e510 con 0x7fdb840a1f90 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.828+0000 7fdb88897700 10 monclient(hunting): authenticate will time out at 1264.188913s 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:40196/0 (socket says 172.21.15.12:40196) 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 -- 172.21.15.12:0/764755797 learned_addr learned my addr 172.21.15.12:0/764755797 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): get_auth_request con 0x7fdb84138980 auth_method 0 2023-09-29T23:15:54.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:54.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb81d9b700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb840a2360 0x7fdb84097900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb8259c700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 0x7fdb84142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb81d9b700 10 monclient(hunting): get_auth_request con 0x7fdb840a2360 auth_method 0 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb81d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb81d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:54.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb81d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient(hunting): handle_auth_done global_id 4612 payload 824 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 -- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 msgr2=0x7fdb84142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 0x7fdb84142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 -- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb840a2360 msgr2=0x7fdb84097900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb840a2360 0x7fdb84097900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:54.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 monclient: found mon.noname-c 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 -- 172.21.15.12:0/764755797 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780094f0 con 0x7fdb84138980 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb8259c700 10 monclient: get_auth_request con 0x7fdb840a1f90 auth_method 0 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb8259c700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 0x7fdb84142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb82d9d700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009df0 tx=0x7fdb7c00b040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb7c014480 con 0x7fdb84138980 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:15:54.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: dump: 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:54.860 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: _finish_auth 0 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 5 monclient: authenticate success, global_id 4612 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c00f040 con 0x7fdb84138980 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 10 monclient: _renew_subs 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb7c014aa0 con 0x7fdb84138980 2023-09-29T23:15:54.861 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8408eb70 con 0x7fdb84138980 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb80d99700 4 set_mon_vals no callback set 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb727fc700 10 monclient: dump: 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:54.862 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 10 monclient: _renew_subs 2023-09-29T23:15:54.863 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.829+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8408df30 con 0x7fdb84138980 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fdb7c014c00 con 0x7fdb84138980 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb88897700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb88897700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb88897700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb84070bc0 con 0x7fdb84138980 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb727fc700 1 --2- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 0x7fdb64038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdb7c047c80 con 0x7fdb84138980 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb8259c700 1 --2- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 0x7fdb64038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.830+0000 7fdb8259c700 10 monclient: get_auth_request con 0x7fdb64036850 auth_method 0 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.831+0000 7fdb8259c700 1 --2- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 0x7fdb64038d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb74000c00 tx=0x7fdb74009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:54.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.835+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb7c014900 con 0x7fdb84138980 2023-09-29T23:15:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.835+0000 7fdb727fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:54.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:54.835+0000 7fdb727fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.070+0000 7fdb88897700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.070+0000 7fdb88897700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.070+0000 7fdb88897700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.070+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdb84066790 con 0x7fdb84138980 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.071+0000 7fdb727fc700 1 -- 172.21.15.12:0/764755797 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdb7c018070 con 0x7fdb84138980 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.071+0000 7fdb727fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.071+0000 7fdb727fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:55.072 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:55.072 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 msgr2=0x7fdb64038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 0x7fdb64038d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb74000c00 tx=0x7fdb74009040 comp rx=0 tx=0).stop 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 10 monclient: shutdown 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 msgr2=0x7fdb84097e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009df0 tx=0x7fdb7c00b040 comp rx=0 tx=0).stop 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 shutdown_connections 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> v2:172.21.15.153:6812/83257 conn(0x7fdb64036850 0x7fdb64038d00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdb84138980 0x7fdb84097e40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:55.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdb840a1f90 0x7fdb84142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 --2- 172.21.15.12:0/764755797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb840a2360 0x7fdb84097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 >> 172.21.15.12:0/764755797 conn(0x7fdb840a3b50 msgr2=0x7fdb8409a660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 shutdown_connections 2023-09-29T23:15:55.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:55.073+0000 7fdb88897700 1 -- 172.21.15.12:0/764755797 wait complete. 2023-09-29T23:15:58.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.214+0000 7f2124de8700 10 monclient: get_monmap_and_config 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.214+0000 7f2124de8700 10 monclient: build_initial_monmap 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.214+0000 7f2124de8700 10 monclient: monmap: 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:15:58.215735+0000 2023-09-29T23:15:58.215 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:15:58.215735+0000 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:58.216 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:58.217 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:15:58.217 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:15:58.217 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.215+0000 7f2124de8700 1 Processor -- start 2023-09-29T23:15:58.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.215+0000 7f2124de8700 1 -- start start 2023-09-29T23:15:58.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.215+0000 7f2124de8700 10 monclient: init 2023-09-29T23:15:58.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:58.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient: _add_conns ranks=[6,1,8,5,9,7,0,4,10,3,2] 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-d con 0x7f2120133130 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2120133500 addr v1:172.21.15.153:6789/0 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 0x7f212013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-e con 0x7f2120138670 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2120068ed0 con 0x7f2120133500 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f212009b2a0 con 0x7f2120133130 2023-09-29T23:15:58.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f21200a27a0 con 0x7f2120138670 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f2124de8700 10 monclient(hunting): authenticate will time out at 1267.576361s 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 0x7f212013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:49926/0 (socket says 172.21.15.12:49926) 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 -- 172.21.15.12:0/11224147 learned_addr learned my addr 172.21.15.12:0/11224147 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): get_auth_request con 0x7f2120133130 auth_method 0 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): get_auth_request con 0x7f2120138670 auth_method 0 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient(hunting): handle_auth_done global_id 4623 payload 274 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 -- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 msgr2=0x7f212013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 0x7f212013aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 -- 172.21.15.12:0/11224147 >> v1:172.21.15.153:6789/0 conn(0x7f2120133500 legacy=0x7f2120133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 monclient: found mon.noname-d 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211f7fe700 1 -- 172.21.15.12:0/11224147 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212004c960 con 0x7f2120133130 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.216+0000 7f211ffff700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 0x7f212013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211f7fe700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2110009a10 tx=0x7f2110009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=34f58a3f74b92e7a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 1 -- 172.21.15.12:0/11224147 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f211000bca0 con 0x7f2120133130 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: dump: 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:58.232 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: _finish_auth 0 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 5 monclient: authenticate success, global_id 4623 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 1 -- 172.21.15.12:0/11224147 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211000be20 con 0x7f2120133130 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 10 monclient: get_monmap_and_config success 2023-09-29T23:15:58.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 4 set_mon_vals no callback set 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 1 -- 172.21.15.12:0/11224147 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f21100118a0 con 0x7f2120133130 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f211effd700 10 monclient: dump: 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.234 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:58.235 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 10 monclient: shutdown 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 -- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 msgr2=0x7f21200a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2110009a10 tx=0x7f2110009250 comp rx=0 tx=0).stop 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 -- 172.21.15.12:0/11224147 shutdown_connections 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f2120138670 0x7f212013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 --2- 172.21.15.12:0/11224147 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120133130 0x7f21200a0e80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 -- 172.21.15.12:0/11224147 >> 172.21.15.12:0/11224147 conn(0x7f21200a6490 msgr2=0x7f21200a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 -- 172.21.15.12:0/11224147 shutdown_connections 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.217+0000 7f2124de8700 1 -- 172.21.15.12:0/11224147 wait complete. 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.218+0000 7f2124de8700 10 monclient: build_initial_monmap 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.218+0000 7f2124de8700 10 monclient: monmap: 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:15:58.237 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:15:58.238 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.218+0000 7f2124de8700 1 Processor -- start 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.218+0000 7f2124de8700 1 -- start start 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.218+0000 7f2124de8700 10 monclient: init 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient: _reopen_session rank -1 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient: _add_conns ranks=[7,4,6,5,1,8,2,0,3] 2023-09-29T23:15:58.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2120133130 0x7f212006e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-i con 0x7f2120133130 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f2120133500 0x7f212009ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-c con 0x7f2120133500 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): picked mon.noname-d con 0x7f2120138670 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f212009b3e0 con 0x7f2120133500 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:58.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f212009b020 con 0x7f2120138670 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f21200a0290 con 0x7f2120133130 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): _renew_subs 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2124de8700 10 monclient(hunting): authenticate will time out at 1267.579327s 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:49942/0 (socket says 172.21.15.12:49942) 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 -- 172.21.15.12:0/880149430 learned_addr learned my addr 172.21.15.12:0/880149430 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): get_auth_request con 0x7f2120138670 auth_method 0 2023-09-29T23:15:58.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f2120133500 0x7f212009ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): get_auth_request con 0x7f2120133500 auth_method 0 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2120133130 0x7f212006e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): get_auth_request con 0x7f2120133130 auth_method 0 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f2117fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:58.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 -- 172.21.15.12:0/880149430 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2120133130 msgr2=0x7f212006e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2120133130 0x7f212006e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 -- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f2120133500 msgr2=0x7f212009ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f2120133500 0x7f212009ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 monclient: found mon.noname-d 2023-09-29T23:15:58.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 -- 172.21.15.12:0/880149430 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2118009a20 con 0x7f2120138670 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.219+0000 7f211ffff700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2118009f60 tx=0x7f211800b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2118014480 con 0x7f2120138670 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: dump: 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:58.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:58.246 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: _finish_auth 0 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 5 monclient: authenticate success, global_id 4632 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211800f040 con 0x7f2120138670 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 10 monclient: _renew_subs 2023-09-29T23:15:58.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21100094f0 con 0x7f2120138670 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2118014aa0 con 0x7f2120138670 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f211e7fc700 4 set_mon_vals no callback set 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 10 monclient: dump: 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:15:58.248 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:15:58.249 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 10 monclient: _renew_subs 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212006ba20 con 0x7f2120138670 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.220+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2118014c00 con 0x7f2120138670 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f2124de8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:58.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f2124de8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f21177fe700 1 --2- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 0x7f2100038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f2124de8700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212012efb0 con 0x7f2120138670 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2118047d20 con 0x7f2120138670 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f211f7fe700 1 --2- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 0x7f2100038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f211f7fe700 10 monclient: get_auth_request con 0x7f2100036850 auth_method 0 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.221+0000 7f211f7fe700 1 --2- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 0x7f2100038d00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2110009fd0 tx=0x7f2110017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.226+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2118018070 con 0x7f2120138670 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.226+0000 7f21177fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:15:58.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.226+0000 7f21177fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:15:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f2124de8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:15:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f2124de8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f2124de8700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:15:58.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2120070bc0 con 0x7f2120138670 2023-09-29T23:15:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f21177fe700 1 -- 172.21.15.12:0/880149430 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f211802a100 con 0x7f2120138670 2023-09-29T23:15:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f21177fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:15:58.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.461+0000 7f21177fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:15:58.462 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:15:58.462 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:15:58.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 msgr2=0x7f2100038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 0x7f2100038d00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2110009fd0 tx=0x7f2110017000 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 10 monclient: shutdown 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 msgr2=0x7f212009f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2118009f60 tx=0x7f211800b040 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 shutdown_connections 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> v2:172.21.15.153:6812/83257 conn(0x7f2100036850 0x7f2100038d00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f2120138670 0x7f212009f3a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f2120133130 0x7f212006e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 --2- 172.21.15.12:0/880149430 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f2120133500 0x7f212009ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 >> 172.21.15.12:0/880149430 conn(0x7f21200a6490 msgr2=0x7f212012e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:15:58.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 shutdown_connections 2023-09-29T23:15:58.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:15:58.463+0000 7f2124de8700 1 -- 172.21.15.12:0/880149430 wait complete. 2023-09-29T23:16:01.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.604+0000 7f3bd4a40700 10 monclient: get_monmap_and_config 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.604+0000 7f3bd4a40700 10 monclient: build_initial_monmap 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.604+0000 7f3bd4a40700 10 monclient: monmap: 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:01.605720+0000 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:01.605720+0000 2023-09-29T23:16:01.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:01.606 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:01.607 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.605+0000 7f3bd4a40700 1 Processor -- start 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.605+0000 7f3bd4a40700 1 -- start start 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.605+0000 7f3bd4a40700 10 monclient: init 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient: _add_conns ranks=[0,5,4,6,1,7,3,8,10,2,9] 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bd0133130 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3bd0133c10 0x7f3bd00a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-h con 0x7f3bd0133c10 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 0x7f3bd013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-c con 0x7f3bd0138670 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd0068ed0 con 0x7f3bd0133130 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd009b2a0 con 0x7f3bd0138670 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00a27a0 con 0x7f3bd0133c10 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bd4a40700 10 monclient(hunting): authenticate will time out at 1270.966155s 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcf7fe700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3bd0133130 0x7f3bd0133500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:39046/0 (socket says 172.21.15.12:39046) 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcf7fe700 1 -- 172.21.15.12:0/1059110223 learned_addr learned my addr 172.21.15.12:0/1059110223 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 0x7f3bd013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): get_auth_request con 0x7f3bd0138670 auth_method 0 2023-09-29T23:16:01.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bceffd700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3bd0133c10 0x7f3bd00a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bceffd700 10 monclient(hunting): get_auth_request con 0x7f3bd0133c10 auth_method 0 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bceffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 1 -- 172.21.15.12:0/1059110223 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918663202 0 0) 0x7f3bd0068ed0 con 0x7f3bd0133130 2023-09-29T23:16:01.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 10 monclient(hunting): my global_id is 4689 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 1 -- 172.21.15.12:0/1059110223 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bbc003570 con 0x7f3bd0133130 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient(hunting): handle_auth_done global_id 4630 payload 274 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 -- 172.21.15.12:0/1059110223 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3bd0133c10 msgr2=0x7f3bd00a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3bd0133c10 0x7f3bd00a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 -- 172.21.15.12:0/1059110223 >> v1:172.21.15.12:6789/0 conn(0x7f3bd0133130 legacy=0x7f3bd0133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient: _finish_hunting 0 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 monclient: found mon.noname-c 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:01.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 -- 172.21.15.12:0/1059110223 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd004c960 con 0x7f3bd0138670 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bcffff700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 0x7f3bd013aa50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009940 tx=0x7f3bc80091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a2405d8ed67daa4f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 1 -- 172.21.15.12:0/1059110223 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bc800baf0 con 0x7f3bd0138670 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.606+0000 7f3bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: dump: 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:01.619 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:01.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 1 -- 172.21.15.12:0/1059110223 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc800bc70 con 0x7f3bd0138670 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 5 monclient: authenticate success, global_id 4630 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 1 -- 172.21.15.12:0/1059110223 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bc800bde0 con 0x7f3bd0138670 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 4 set_mon_vals no callback set 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bce7fc700 10 monclient: dump: 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:01.621 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:01.622 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 10 monclient: shutdown 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 -- 172.21.15.12:0/1059110223 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 msgr2=0x7f3bd013aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 0x7f3bd013aa50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009940 tx=0x7f3bc80091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 -- 172.21.15.12:0/1059110223 shutdown_connections 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f3bd0138670 0x7f3bd013aa50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/1059110223 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3bd0133c10 0x7f3bd00a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 -- 172.21.15.12:0/1059110223 >> 172.21.15.12:0/1059110223 conn(0x7f3bd00a6490 msgr2=0x7f3bd00a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 -- 172.21.15.12:0/1059110223 shutdown_connections 2023-09-29T23:16:01.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.607+0000 7f3bd4a40700 1 -- 172.21.15.12:0/1059110223 wait complete. 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.608+0000 7f3bd4a40700 10 monclient: build_initial_monmap 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.608+0000 7f3bd4a40700 10 monclient: monmap: 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:01.624 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.608+0000 7f3bd4a40700 1 Processor -- start 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.608+0000 7f3bd4a40700 1 -- start start 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.608+0000 7f3bd4a40700 10 monclient: init 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient: _add_conns ranks=[2,6,0,8,1,4,7,5,3] 2023-09-29T23:16:01.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 0x7f3bd01429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-b con 0x7f3bd0133130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3bd0133c10 0x7f3bd0142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-d con 0x7f3bd0133c10 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): picked mon.noname-a con 0x7f3bd0138670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00a06c0 con 0x7f3bd0138670 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd0142640 con 0x7f3bd0133130 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd009b020 con 0x7f3bd0133c10 2023-09-29T23:16:01.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bd4a40700 10 monclient(hunting): authenticate will time out at 1270.969173s 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 0x7f3bd01429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3bd0133c10 0x7f3bd0142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34238/0 (socket says 172.21.15.12:34238) 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 -- 172.21.15.12:0/3343979966 learned_addr learned my addr 172.21.15.12:0/3343979966 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): get_auth_request con 0x7f3bd0138670 auth_method 0 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): get_auth_request con 0x7f3bd0133c10 auth_method 0 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): get_auth_request con 0x7f3bd0133130 auth_method 0 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:01.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient(hunting): handle_auth_done global_id 4698 payload 824 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 -- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3bd0133c10 msgr2=0x7f3bd0142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3bd0133c10 0x7f3bd0142100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 -- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 msgr2=0x7f3bd01429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 0x7f3bd01429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient: _finish_hunting 0 2023-09-29T23:16:01.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 monclient: found mon.noname-a 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 -- 172.21.15.12:0/3343979966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc8009420 con 0x7f3bd0138670 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcf7fe700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 0x7f3bd01429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bcffff700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3bd009c600 tx=0x7f3bc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.609+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bc8006ab0 con 0x7f3bd0138670 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: dump: 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:01.630 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:01.631 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: _finish_auth 0 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc801bda0 con 0x7f3bd0138670 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 5 monclient: authenticate success, global_id 4698 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 10 monclient: _renew_subs 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd01344a0 con 0x7f3bd0138670 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:01.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3bc80118a0 con 0x7f3bd0138670 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 10 monclient: dump: 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:01.633 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bcdffb700 4 set_mon_vals no callback set 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 10 monclient: _renew_subs 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd0136010 con 0x7f3bd0138670 2023-09-29T23:16:01.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.610+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3bc8023030 con 0x7f3bd0138670 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bd4a40700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bd4a40700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bd4a40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd0070bc0 con 0x7f3bd0138670 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bc77fe700 1 --2- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 0x7f3bb0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3bc8048880 con 0x7f3bd0138670 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bcf7fe700 1 --2- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 0x7f3bb0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bcf7fe700 10 monclient: get_auth_request con 0x7f3bb0036850 auth_method 0 2023-09-29T23:16:01.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.611+0000 7f3bcf7fe700 1 --2- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 0x7f3bb0038d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0005c50 tx=0x7f3bc0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.615+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bc801b920 con 0x7f3bd0138670 2023-09-29T23:16:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.615+0000 7f3bc77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:01.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.615+0000 7f3bc77fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:01.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.853+0000 7f3bd4a40700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:01.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.853+0000 7f3bd4a40700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:01.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.853+0000 7f3bd4a40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:01.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.854+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3bd0134b60 con 0x7f3bd0138670 2023-09-29T23:16:01.854 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.854+0000 7f3bc77fe700 1 -- 172.21.15.12:0/3343979966 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3bc8015070 con 0x7f3bd0138670 2023-09-29T23:16:01.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.854+0000 7f3bc77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:01.855 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.854+0000 7f3bc77fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:01.855 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:01.855 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:01.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 msgr2=0x7f3bb0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.856 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 0x7f3bb0038d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0005c50 tx=0x7f3bc0009040 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 10 monclient: shutdown 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 msgr2=0x7f3bd0137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3bd009c600 tx=0x7f3bc8017000 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 shutdown_connections 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bd0138670 0x7f3bd0137820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> v2:172.21.15.153:6812/83257 conn(0x7f3bb0036850 0x7f3bb0038d00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3bd0133130 0x7f3bd01429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 --2- 172.21.15.12:0/3343979966 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3bd0133c10 0x7f3bd0142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 >> 172.21.15.12:0/3343979966 conn(0x7f3bd00a6490 msgr2=0x7f3bd00a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:01.857 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 shutdown_connections 2023-09-29T23:16:01.858 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:01.856+0000 7f3bd4a40700 1 -- 172.21.15.12:0/3343979966 wait complete. 2023-09-29T23:16:04.868 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:04.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.998+0000 7f3e8f6bb700 10 monclient: get_monmap_and_config 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.998+0000 7f3e8f6bb700 10 monclient: build_initial_monmap 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.998+0000 7f3e8f6bb700 10 monclient: monmap: 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:04.999499+0000 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:04.999499+0000 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:04.999 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 Processor -- start 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 -- start start 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient: init 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient: _add_conns ranks=[0,6,7,9,10,1,3,4,5,2,8] 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e880a1f90 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 0x7f3e880a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-d con 0x7f3e880a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 0x7f3e8813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-i con 0x7f3e88138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e88068ed0 con 0x7f3e880a1f90 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8809e350 con 0x7f3e880a2360 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8809fcf0 con 0x7f3e88138980 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:04.999+0000 7f3e8f6bb700 10 monclient(hunting): authenticate will time out at 1274.359954s 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 0x7f3e880a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8d457700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3e880a1f90 0x7f3e8809f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48912/0 (socket says 172.21.15.12:48912) 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8d457700 1 -- 172.21.15.12:0/2605043248 learned_addr learned my addr 172.21.15.12:0/2605043248 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): get_auth_request con 0x7f3e880a2360 auth_method 0 2023-09-29T23:16:05.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 0x7f3e8813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 10 monclient(hunting): get_auth_request con 0x7f3e88138980 auth_method 0 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:05.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_done global_id 4650 payload 274 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 -- 172.21.15.12:0/2605043248 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 msgr2=0x7f3e8813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 1 -- 172.21.15.12:0/2605043248 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511723215 0 0) 0x7f3e88068ed0 con 0x7f3e880a1f90 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 0x7f3e8813ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 -- 172.21.15.12:0/2605043248 >> v1:172.21.15.12:6789/0 conn(0x7f3e880a1f90 legacy=0x7f3e8809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient: _finish_hunting 0 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 monclient: found mon.noname-d 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 -- 172.21.15.12:0/2605043248 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8804c960 con 0x7f3e880a2360 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8dc58700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 0x7f3e8813ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:16:05.012 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8cc56700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 0x7f3e880a27b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009910 tx=0x7f3e840091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=aa3be10f597cb8ca server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 1 -- 172.21.15.12:0/2605043248 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3e8400bc80 con 0x7f3e880a2360 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: dump: 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:05.013 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: _finish_auth 0 2023-09-29T23:16:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 1 -- 172.21.15.12:0/2605043248 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8400be00 con 0x7f3e880a2360 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8f6bb700 5 monclient: authenticate success, global_id 4650 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8f6bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 1 -- 172.21.15.12:0/2605043248 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3e840118a0 con 0x7f3e880a2360 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8f6bb700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e8f6bb700 4 set_mon_vals no callback set 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.000+0000 7f3e7ffff700 10 monclient: dump: 2023-09-29T23:16:05.015 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:05.016 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 10 monclient: shutdown 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/2605043248 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 msgr2=0x7f3e880a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 0x7f3e880a27b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009910 tx=0x7f3e840091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/2605043248 shutdown_connections 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3e88138980 0x7f3e8813ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/2605043248 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3e880a2360 0x7f3e880a27b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/2605043248 >> 172.21.15.12:0/2605043248 conn(0x7f3e880a3b50 msgr2=0x7f3e880a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/2605043248 shutdown_connections 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/2605043248 wait complete. 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 10 monclient: build_initial_monmap 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.001+0000 7f3e8f6bb700 10 monclient: monmap: 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:05.018 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 1 Processor -- start 2023-09-29T23:16:05.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 1 -- start start 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 10 monclient: init 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 10 monclient: _add_conns ranks=[0,8,3,6,7,5,1,2,4] 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-a con 0x7f3e880a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 0x7f3e881420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.002+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-e con 0x7f3e880a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 0x7f3e8808c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): picked mon.noname-g con 0x7f3e88138980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8808fdc0 con 0x7f3e880a1f90 2023-09-29T23:16:05.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8808ecf0 con 0x7f3e88138980 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8809e510 con 0x7f3e880a2360 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient(hunting): authenticate will time out at 1274.363039s 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 0x7f3e881420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36892/0 (socket says 172.21.15.12:36892) 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 -- 172.21.15.12:0/3109123815 learned_addr learned my addr 172.21.15.12:0/3109123815 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): get_auth_request con 0x7f3e880a1f90 auth_method 0 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:05.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): get_auth_request con 0x7f3e880a2360 auth_method 0 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:05.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 0x7f3e8808c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 10 monclient(hunting): get_auth_request con 0x7f3e88138980 auth_method 0 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient(hunting): handle_auth_done global_id 4716 payload 824 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 -- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 msgr2=0x7f3e881420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 0x7f3e881420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 -- 172.21.15.12:0/3109123815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 msgr2=0x7f3e8808c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 0x7f3e8808c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient: _finish_hunting 0 2023-09-29T23:16:05.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 monclient: found mon.noname-a 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 -- 172.21.15.12:0/3109123815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84009420 con 0x7f3e880a1f90 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8cc56700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 0x7f3e881420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8dc58700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 0x7f3e8808c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8d457700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800c870 tx=0x7f3e78007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3e780070d0 con 0x7f3e880a1f90 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: dump: 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:05.024 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:05.025 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e78007250 con 0x7f3e880a1f90 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 5 monclient: authenticate success, global_id 4716 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3e78011360 con 0x7f3e880a1f90 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient: _renew_subs 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e88142fd0 con 0x7f3e880a1f90 2023-09-29T23:16:05.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7f7fe700 4 set_mon_vals no callback set 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e7d7fa700 10 monclient: dump: 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:05.027 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient: _renew_subs 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.003+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8809ffc0 con 0x7f3e880a1f90 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.004+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3e78005b30 con 0x7f3e880a1f90 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.004+0000 7f3e7d7fa700 1 --2- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 0x7f3e6c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:05.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.004+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3e78046240 con 0x7f3e880a1f90 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8f6bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8f6bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8f6bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e88142fd0 con 0x7f3e880a1f90 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8cc56700 1 --2- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 0x7f3e6c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8cc56700 10 monclient: get_auth_request con 0x7f3e6c036850 auth_method 0 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.005+0000 7f3e8cc56700 1 --2- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 0x7f3e6c038d00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009910 tx=0x7f3e84017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:05.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.009+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e88142fd0 con 0x7f3e880a1f90 2023-09-29T23:16:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.009+0000 7f3e7d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:05.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.009+0000 7f3e7d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.244+0000 7f3e8f6bb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.244+0000 7f3e8f6bb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.244+0000 7f3e8f6bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.244+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e88142fd0 con 0x7f3e880a1f90 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.245+0000 7f3e7d7fa700 1 -- 172.21.15.12:0/3109123815 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3e88142fd0 con 0x7f3e880a1f90 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.245+0000 7f3e7d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.245+0000 7f3e7d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:05.246 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 msgr2=0x7f3e6c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 0x7f3e6c038d00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009910 tx=0x7f3e84017000 comp rx=0 tx=0).stop 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 10 monclient: shutdown 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 msgr2=0x7f3e88097910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.246+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800c870 tx=0x7f3e78007d90 comp rx=0 tx=0).stop 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 shutdown_connections 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> v2:172.21.15.153:6812/83257 conn(0x7f3e6c036850 0x7f3e6c038d00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e88138980 0x7f3e8808c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3e880a1f90 0x7f3e88097910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 --2- 172.21.15.12:0/3109123815 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f3e880a2360 0x7f3e881420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 >> 172.21.15.12:0/3109123815 conn(0x7f3e880a3b50 msgr2=0x7f3e8809a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 shutdown_connections 2023-09-29T23:16:05.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:05.247+0000 7f3e8f6bb700 1 -- 172.21.15.12:0/3109123815 wait complete. 2023-09-29T23:16:08.259 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.387+0000 7f343a1d3700 10 monclient: get_monmap_and_config 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.387+0000 7f343a1d3700 10 monclient: build_initial_monmap 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.387+0000 7f343a1d3700 10 monclient: monmap: 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:08.388614+0000 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:08.388614+0000 2023-09-29T23:16:08.388 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:08.389 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:08.390 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.388+0000 7f343a1d3700 1 Processor -- start 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.388+0000 7f343a1d3700 1 -- start start 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.388+0000 7f343a1d3700 10 monclient: init 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.388+0000 7f343a1d3700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient: _add_conns ranks=[4,0,3,7,1,2,5,9,10,8,6] 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-c con 0x7f343406b770 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f343406bb40 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:08.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f3434097500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-g con 0x7f34340970f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3434068ed0 con 0x7f343406bb40 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343406bf10 con 0x7f34340970f0 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f343409f9e0 con 0x7f343406b770 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient(hunting): authenticate will time out at 1277.749103s 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:51280/0 (socket says 172.21.15.12:51280) 2023-09-29T23:16:08.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 -- 172.21.15.12:0/831631316 learned_addr learned my addr 172.21.15.12:0/831631316 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): get_auth_request con 0x7f343406b770 auth_method 0 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient(hunting): handle_auth_done global_id 4639 payload 274 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34399d2700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f3434097500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 1 -- 172.21.15.12:0/831631316 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908427166 0 0) 0x7f3434068ed0 con 0x7f343406bb40 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 -- 172.21.15.12:0/831631316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 msgr2=0x7f3434097500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:08.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f3434097500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 -- 172.21.15.12:0/831631316 >> v1:172.21.15.12:6789/0 conn(0x7f343406bb40 legacy=0x7f34340969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient: _finish_hunting 0 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 monclient: found mon.noname-c 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 -- 172.21.15.12:0/831631316 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343404c960 con 0x7f343406b770 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34399d2700 10 monclient: get_auth_request con 0x7f34340970f0 auth_method 0 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34399d2700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f3434097500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f34391d1700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34300099e0 tx=0x7f3430009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=43f3ddaf6eefede2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:08.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 1 -- 172.21.15.12:0/831631316 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f343000bce0 con 0x7f343406b770 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: dump: 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:08.402 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: _finish_auth 0 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 5 monclient: authenticate success, global_id 4639 2023-09-29T23:16:08.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 1 -- 172.21.15.12:0/831631316 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000be60 con 0x7f343406b770 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f343a1d3700 4 set_mon_vals no callback set 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 1 -- 172.21.15.12:0/831631316 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f34300118a0 con 0x7f343406b770 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.389+0000 7f342bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f342bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f342bfff700 10 monclient: dump: 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:08.404 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 10 monclient: shutdown 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 -- 172.21.15.12:0/831631316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 msgr2=0x7f343406c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34300099e0 tx=0x7f3430009250 comp rx=0 tx=0).stop 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 -- 172.21.15.12:0/831631316 shutdown_connections 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f3434097500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 --2- 172.21.15.12:0/831631316 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f343406b770 0x7f343406c260 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 -- 172.21.15.12:0/831631316 >> 172.21.15.12:0/831631316 conn(0x7f3434063e30 msgr2=0x7f34340a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 -- 172.21.15.12:0/831631316 shutdown_connections 2023-09-29T23:16:08.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.390+0000 7f343a1d3700 1 -- 172.21.15.12:0/831631316 wait complete. 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.391+0000 7f343a1d3700 10 monclient: build_initial_monmap 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.391+0000 7f343a1d3700 10 monclient: monmap: 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:08.407 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.391+0000 7f343a1d3700 1 Processor -- start 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.391+0000 7f343a1d3700 1 -- start start 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.391+0000 7f343a1d3700 10 monclient: init 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient: _add_conns ranks=[2,6,3,7,1,4,5,0,8] 2023-09-29T23:16:08.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-b con 0x7f343406b770 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f343406bb40 0x7f3434142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-d con 0x7f343406bb40 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f34341465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): picked mon.noname-g con 0x7f34340970f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343406f790 con 0x7f343406b770 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343411d1c0 con 0x7f34340970f0 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3434143bc0 con 0x7f343406bb40 2023-09-29T23:16:08.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f343a1d3700 10 monclient(hunting): authenticate will time out at 1277.752344s 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f343406bb40 0x7f3434142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:36496/0 (socket says 172.21.15.12:36496) 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 -- 172.21.15.12:0/1922395113 learned_addr learned my addr 172.21.15.12:0/1922395113 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): get_auth_request con 0x7f343406b770 auth_method 0 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): get_auth_request con 0x7f343406bb40 auth_method 0 2023-09-29T23:16:08.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34389d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34399d2700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f34341465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34399d2700 10 monclient(hunting): get_auth_request con 0x7f34340970f0 auth_method 0 2023-09-29T23:16:08.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34399d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34399d2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34399d2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient(hunting): handle_auth_done global_id 4520 payload 824 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 -- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f343406bb40 msgr2=0x7f3434142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f343406bb40 0x7f3434142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 -- 172.21.15.12:0/1922395113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 msgr2=0x7f34341465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f34341465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient: _finish_hunting 0 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 monclient: found mon.noname-b 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:16:08.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 -- 172.21.15.12:0/1922395113 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300094f0 con 0x7f343406b770 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.392+0000 7f34391d1700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3430006a90 con 0x7f343406b770 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: dump: 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:08.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:08.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 5 monclient: authenticate success, global_id 4520 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34300118a0 con 0x7f343406b770 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3430011e90 con 0x7f343406b770 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 10 monclient: _renew_subs 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f342b7fe700 4 set_mon_vals no callback set 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34341277d0 con 0x7f343406b770 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f34297fa700 10 monclient: dump: 2023-09-29T23:16:08.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:08.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 10 monclient: _renew_subs 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.393+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343411dd70 con 0x7f343406b770 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3430011a10 con 0x7f343406b770 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f343a1d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f343a1d3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f343a1d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343411df00 con 0x7f343406b770 2023-09-29T23:16:08.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34297fa700 1 --2- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 0x7f3418038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3430048cd0 con 0x7f343406b770 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34389d0700 1 --2- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 0x7f3418038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34389d0700 10 monclient: get_auth_request con 0x7f3418036850 auth_method 0 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.394+0000 7f34389d0700 1 --2- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 0x7f3418038d00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f343411b450 tx=0x7f3424009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.398+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3430011420 con 0x7f343406b770 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.398+0000 7f34297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:08.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.398+0000 7f34297fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f343a1d3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f343a1d3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f343a1d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:08.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3434070bc0 con 0x7f343406b770 2023-09-29T23:16:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f34297fa700 1 -- 172.21.15.12:0/1922395113 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3430015070 con 0x7f343406b770 2023-09-29T23:16:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f34297fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:08.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.633+0000 7f34297fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:08.634 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:08.634 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 msgr2=0x7f3418038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 0x7f3418038d00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f343411b450 tx=0x7f3424009040 comp rx=0 tx=0).stop 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 10 monclient: shutdown 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 msgr2=0x7f3434143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:08.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430017000 comp rx=0 tx=0).stop 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 shutdown_connections 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> v2:172.21.15.153:6812/83257 conn(0x7f3418036850 0x7f3418038d00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34340970f0 0x7f34341465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f343406b770 0x7f3434143060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 --2- 172.21.15.12:0/1922395113 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f343406bb40 0x7f3434142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 >> 172.21.15.12:0/1922395113 conn(0x7f3434063e30 msgr2=0x7f34340a2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 shutdown_connections 2023-09-29T23:16:08.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:08.635+0000 7f343a1d3700 1 -- 172.21.15.12:0/1922395113 wait complete. 2023-09-29T23:16:11.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 10 monclient: get_monmap_and_config 2023-09-29T23:16:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 10 monclient: build_initial_monmap 2023-09-29T23:16:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 10 monclient: monmap: 2023-09-29T23:16:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:11.776 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:11.777133+0000 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:11.777133+0000 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:11.777 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:11.778 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:11.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 1 Processor -- start 2023-09-29T23:16:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 1 -- start start 2023-09-29T23:16:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.776+0000 7f957c955700 10 monclient: init 2023-09-29T23:16:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:11.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient: _add_conns ranks=[4,5,9,1,7,8,10,2,3,0,6] 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): picked mon.noname-c con 0x7f95780a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f95780a2360 0x7f95780a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): picked mon.noname-h con 0x7f95780a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9578133740 0x7f9578133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): picked mon.noname-a con 0x7f9578133740 addr v2:172.21.15.12:3300/0 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957809e350 con 0x7f9578133740 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.788 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f957809ef70 con 0x7f95780a1f90 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f957809da70 con 0x7f95780a2360 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957c955700 10 monclient(hunting): authenticate will time out at 1281.137713s 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9578133740 0x7f9578133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:51292/0 (socket says 172.21.15.12:51292) 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 1 -- 172.21.15.12:0/3802199880 learned_addr learned my addr 172.21.15.12:0/3802199880 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): get_auth_request con 0x7f95780a1f90 auth_method 0 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): get_auth_request con 0x7f9578133740 auth_method 0 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.777+0000 7f9576d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:11.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9575d9b700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f95780a2360 0x7f95780a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9575d9b700 10 monclient(hunting): get_auth_request con 0x7f95780a2360 auth_method 0 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9575d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9575d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9575d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 -- 172.21.15.12:0/3802199880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f95780a2360 msgr2=0x7f95780a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f95780a2360 0x7f95780a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 -- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 msgr2=0x7f957809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:16:11.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 monclient: found mon.noname-a 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 -- 172.21.15.12:0/3802199880 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9578068ed0 con 0x7f9578133740 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957659c700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f9576d9d700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9578133740 0x7f9578133bb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9570009e90 tx=0x7f9570009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc67b6fc28d1f44 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 1 -- 172.21.15.12:0/3802199880 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f957000e070 con 0x7f9578133740 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: dump: 2023-09-29T23:16:11.792 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:11.793 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: _finish_auth 0 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 5 monclient: authenticate success, global_id 4734 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 1 -- 172.21.15.12:0/3802199880 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957000bbf0 con 0x7f9578133740 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:11.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 1 -- 172.21.15.12:0/3802199880 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f957000bd60 con 0x7f9578133740 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 4 set_mon_vals no callback set 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957559a700 10 monclient: dump: 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:11.795 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 10 monclient: shutdown 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:11.796 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 1 -- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9578133740 msgr2=0x7f9578133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.778+0000 7f957c955700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9578133740 0x7f9578133bb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9570009e90 tx=0x7f9570009780 comp rx=0 tx=0).stop 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 -- 172.21.15.12:0/3802199880 shutdown_connections 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9578133740 0x7f9578133bb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f95780a1f90 0x7f957809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 --2- 172.21.15.12:0/3802199880 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f95780a2360 0x7f95780a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 -- 172.21.15.12:0/3802199880 >> 172.21.15.12:0/3802199880 conn(0x7f95780a3b50 msgr2=0x7f95780a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 -- 172.21.15.12:0/3802199880 shutdown_connections 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 -- 172.21.15.12:0/3802199880 wait complete. 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 10 monclient: build_initial_monmap 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 10 monclient: monmap: 2023-09-29T23:16:11.797 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:11.798 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 Processor -- start 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 1 -- start start 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.779+0000 7f957c955700 10 monclient: init 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient: _add_conns ranks=[8,2,7,6,3,5,0,4,1] 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): picked mon.noname-e con 0x7f95780a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:11.799 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): picked mon.noname-b con 0x7f95780a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9578133740 0x7f957812d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): picked mon.noname-i con 0x7f9578133740 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9578130b60 con 0x7f95780a2360 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9578130e40 con 0x7f9578133740 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f9578131120 con 0x7f95780a1f90 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957c955700 10 monclient(hunting): authenticate will time out at 1281.140432s 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.800 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:48384/0 (socket says 172.21.15.12:48384) 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:36504/0 (socket says 172.21.15.12:36504) 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 -- 172.21.15.12:0/54572653 learned_addr learned my addr 172.21.15.12:0/54572653 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): get_auth_request con 0x7f95780a1f90 auth_method 0 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): get_auth_request con 0x7f95780a2360 auth_method 0 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9576d9d700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9578133740 0x7f957812d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9576d9d700 10 monclient(hunting): get_auth_request con 0x7f9578133740 auth_method 0 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9576d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:11.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9576d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9576d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 -- 172.21.15.12:0/54572653 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9578133740 msgr2=0x7f957812d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9578133740 0x7f957812d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 -- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 msgr2=0x7f9578132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient: _finish_hunting 0 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 monclient: found mon.noname-e 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:11.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f957659c700 1 -- 172.21.15.12:0/54572653 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95680094f0 con 0x7f95780a1f90 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.780+0000 7f9575d9b700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957659c700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f956800ec50 tx=0x7f9568009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f95680097d0 con 0x7f95780a1f90 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: dump: 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:11.804 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956800cda0 con 0x7f95780a1f90 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 5 monclient: authenticate success, global_id 4508 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 10 monclient: _renew_subs 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9570009a20 con 0x7f95780a1f90 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f95680138f0 con 0x7f95780a1f90 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:11.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f95667fc700 10 monclient: dump: 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:11.807 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f9574d99700 4 set_mon_vals no callback set 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 10 monclient: _renew_subs 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.781+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957804c960 con 0x7f95780a1f90 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9568013470 con 0x7f95780a1f90 2023-09-29T23:16:11.808 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f957c955700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f957c955700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f957c955700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95781312e0 con 0x7f95780a1f90 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f95667fc700 1 --2- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 0x7f9558038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9568047770 con 0x7f95780a1f90 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f9575d9b700 1 --2- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 0x7f9558038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.782+0000 7f9575d9b700 10 monclient: get_auth_request con 0x7f9558036850 auth_method 0 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.783+0000 7f9575d9b700 1 --2- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 0x7f9558038d00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f956c006fd0 tx=0x7f956c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.787+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95680192c0 con 0x7f95780a1f90 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.787+0000 7f95667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:11.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:11.787+0000 7f95667fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f957c955700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f957c955700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f957c955700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f957809f8d0 con 0x7f95780a1f90 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f95667fc700 1 -- 172.21.15.12:0/54572653 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9568011350 con 0x7f95780a1f90 2023-09-29T23:16:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f95667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:12.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.020+0000 7f95667fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:12.021 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:12.021 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 msgr2=0x7f9558038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 0x7f9558038d00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f956c006fd0 tx=0x7f956c008040 comp rx=0 tx=0).stop 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 10 monclient: shutdown 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 msgr2=0x7f9578142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f956800ec50 tx=0x7f9568009250 comp rx=0 tx=0).stop 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 shutdown_connections 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9578133740 0x7f957812d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f95780a1f90 0x7f9578142210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> v2:172.21.15.153:6812/83257 conn(0x7f9558036850 0x7f9558038d00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 --2- 172.21.15.12:0/54572653 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f95780a2360 0x7f9578132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 >> 172.21.15.12:0/54572653 conn(0x7f95780a3b50 msgr2=0x7f95780a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 shutdown_connections 2023-09-29T23:16:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:12.022+0000 7f957c955700 1 -- 172.21.15.12:0/54572653 wait complete. 2023-09-29T23:16:15.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:15.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 10 monclient: get_monmap_and_config 2023-09-29T23:16:15.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 10 monclient: build_initial_monmap 2023-09-29T23:16:15.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 10 monclient: monmap: 2023-09-29T23:16:15.166 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:15.167022+0000 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:15.167022+0000 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:15.167 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:15.168 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:15.168 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:15.168 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:15.168 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:15.168 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 1 Processor -- start 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 1 -- start start 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.166+0000 7f3adef21700 10 monclient: init 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient: _add_conns ranks=[6,0,7,8,9,10,3,4,1,2,5] 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 0x7f3ad809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-d con 0x7f3ad80a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ad80a2360 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3ad8138980 0x7f3ad813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-i con 0x7f3ad8138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:16:15.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad8068ed0 con 0x7f3ad80a2360 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809e350 con 0x7f3ad80a1f90 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809ef70 con 0x7f3ad8138980 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adef21700 10 monclient(hunting): authenticate will time out at 1284.527468s 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 0x7f3ad809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad7fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f3ad80a2360 0x7f3ad80a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36514/0 (socket says 172.21.15.12:36514) 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad7fff700 1 -- 172.21.15.12:0/968012827 learned_addr learned my addr 172.21.15.12:0/968012827 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): get_auth_request con 0x7f3ad80a1f90 auth_method 0 2023-09-29T23:16:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3ad8138980 0x7f3ad813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): get_auth_request con 0x7f3ad8138980 auth_method 0 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad77fe700 1 -- 172.21.15.12:0/968012827 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395422475 0 0) 0x7f3ad8068ed0 con 0x7f3ad80a2360 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad77fe700 10 monclient(hunting): my global_id is 4743 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3ad77fe700 1 -- 172.21.15.12:0/968012827 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac4003570 con 0x7f3ad80a2360 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient(hunting): handle_auth_done global_id 4668 payload 274 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 -- 172.21.15.12:0/968012827 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3ad8138980 msgr2=0x7f3ad813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3ad8138980 0x7f3ad813ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 -- 172.21.15.12:0/968012827 >> v1:172.21.15.12:6789/0 conn(0x7f3ad80a2360 legacy=0x7f3ad80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient: _finish_hunting 0 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 monclient: found mon.noname-d 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.167+0000 7f3adccbd700 1 -- 172.21.15.12:0/968012827 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad804c960 con 0x7f3ad80a1f90 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adccbd700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 0x7f3ad809f5e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009910 tx=0x7f3ac00091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=123acbc05b42bb7f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 1 -- 172.21.15.12:0/968012827 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ac0011ab0 con 0x7f3ad80a1f90 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: dump: 2023-09-29T23:16:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: _finish_auth 0 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 1 -- 172.21.15.12:0/968012827 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0011c30 con 0x7f3ad80a1f90 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 5 monclient: authenticate success, global_id 4668 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:15.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 1 -- 172.21.15.12:0/968012827 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ac0011da0 con 0x7f3ad80a1f90 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 4 set_mon_vals no callback set 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3ad77fe700 10 monclient: dump: 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 10 monclient: shutdown 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 -- 172.21.15.12:0/968012827 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 msgr2=0x7f3ad809f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 0x7f3ad809f5e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009910 tx=0x7f3ac00091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 -- 172.21.15.12:0/968012827 shutdown_connections 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f3ad8138980 0x7f3ad813ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 --2- 172.21.15.12:0/968012827 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3ad80a1f90 0x7f3ad809f5e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 -- 172.21.15.12:0/968012827 >> 172.21.15.12:0/968012827 conn(0x7f3ad80a3b50 msgr2=0x7f3ad80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 -- 172.21.15.12:0/968012827 shutdown_connections 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.168+0000 7f3adef21700 1 -- 172.21.15.12:0/968012827 wait complete. 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.169+0000 7f3adef21700 10 monclient: build_initial_monmap 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.169+0000 7f3adef21700 10 monclient: monmap: 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.169+0000 7f3adef21700 1 Processor -- start 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.169+0000 7f3adef21700 1 -- start start 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.169+0000 7f3adef21700 10 monclient: init 2023-09-29T23:16:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient: _add_conns ranks=[5,1,0,6,2,7,3,4,8] 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3ad80a1f90 0x7f3ad8142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-h con 0x7f3ad80a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ad80a2360 0x7f3ad8137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-f con 0x7f3ad80a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): picked mon.noname-a con 0x7f3ad8138980 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad812fbd0 con 0x7f3ad8138980 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:15.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8143060 con 0x7f3ad80a2360 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809ffc0 con 0x7f3ad80a1f90 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adef21700 10 monclient(hunting): authenticate will time out at 1284.530441s 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3ad80a1f90 0x7f3ad8142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40386/0 (socket says 172.21.15.12:40386) 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ad80a2360 0x7f3ad8137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 -- 172.21.15.12:0/628486261 learned_addr learned my addr 172.21.15.12:0/628486261 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:15.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): get_auth_request con 0x7f3ad8138980 auth_method 0 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): get_auth_request con 0x7f3ad80a2360 auth_method 0 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): get_auth_request con 0x7f3ad80a1f90 auth_method 0 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3adccbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3ad7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient(hunting): handle_auth_done global_id 4752 payload 824 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 -- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3ad80a1f90 msgr2=0x7f3ad8142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3ad80a1f90 0x7f3ad8142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 -- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ad80a2360 msgr2=0x7f3ad8137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ad80a2360 0x7f3ad8137c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient: _finish_hunting 0 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 monclient: found mon.noname-a 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.170+0000 7f3add4be700 1 -- 172.21.15.12:0/628486261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009420 con 0x7f3ad8138980 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3add4be700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000ea80 tx=0x7f3ad00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ad00096b0 con 0x7f3ad8138980 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: dump: 2023-09-29T23:16:15.193 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:15.194 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: _finish_auth 0 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 5 monclient: authenticate success, global_id 4752 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad000ccd0 con 0x7f3ad8138980 2023-09-29T23:16:15.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 10 monclient: _renew_subs 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad809e510 con 0x7f3ad8138980 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ad000ce40 con 0x7f3ad8138980 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad6ffd700 4 set_mon_vals no callback set 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3ad4ff9700 10 monclient: dump: 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.196 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:15.197 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 10 monclient: _renew_subs 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.171+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad8130e60 con 0x7f3ad8138980 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3ad000c850 con 0x7f3ad8138980 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adef21700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adef21700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adef21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3ad4ff9700 1 --2- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 0x7f3ab8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad8070bc0 con 0x7f3ad8138980 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3ad0047470 con 0x7f3ad8138980 2023-09-29T23:16:15.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adccbd700 1 --2- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 0x7f3ab8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:15.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.172+0000 7f3adccbd700 10 monclient: get_auth_request con 0x7f3ab8036850 auth_method 0 2023-09-29T23:16:15.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.173+0000 7f3adccbd700 1 --2- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 0x7f3ab8038d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009910 tx=0x7f3ac0017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:15.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.177+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ad0013470 con 0x7f3ad8138980 2023-09-29T23:16:15.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.177+0000 7f3ad4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:15.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.177+0000 7f3ad4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:15.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.414+0000 7f3adef21700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.414+0000 7f3adef21700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.414+0000 7f3adef21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.414+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ad8131380 con 0x7f3ad8138980 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.415+0000 7f3ad4ff9700 1 -- 172.21.15.12:0/628486261 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3ad001b3b0 con 0x7f3ad8138980 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.415+0000 7f3ad4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.415+0000 7f3ad4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:15.415 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:15.416 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:15.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 msgr2=0x7f3ab8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 0x7f3ab8038d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009910 tx=0x7f3ac0017000 comp rx=0 tx=0).stop 2023-09-29T23:16:15.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 10 monclient: shutdown 2023-09-29T23:16:15.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 msgr2=0x7f3ad81381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000ea80 tx=0x7f3ad00091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 shutdown_connections 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> v2:172.21.15.153:6812/83257 conn(0x7f3ab8036850 0x7f3ab8038d00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3ad8138980 0x7f3ad81381c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3ad80a1f90 0x7f3ad8142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 --2- 172.21.15.12:0/628486261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ad80a2360 0x7f3ad8137c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 >> 172.21.15.12:0/628486261 conn(0x7f3ad80a3b50 msgr2=0x7f3ad809a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 shutdown_connections 2023-09-29T23:16:15.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:15.417+0000 7f3adef21700 1 -- 172.21.15.12:0/628486261 wait complete. 2023-09-29T23:16:18.429 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:18.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.555+0000 7f28a359e700 10 monclient: get_monmap_and_config 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.555+0000 7f28a359e700 10 monclient: build_initial_monmap 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.555+0000 7f28a359e700 10 monclient: monmap: 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:18.556557+0000 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:18.556557+0000 2023-09-29T23:16:18.557 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:18.558 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:18.559 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 Processor -- start 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 -- start start 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient: init 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient: _add_conns ranks=[8,6,9,1,5,0,4,3,7,2,10] 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-e con 0x7f28a4133130 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-d con 0x7f28a4133500 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-a con 0x7f28a412fc20 addr v2:172.21.15.12:3300/0 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40a27a0 con 0x7f28a412fc20 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f28a409adf0 con 0x7f28a4133500 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f28a409bd60 con 0x7f28a4133130 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.556+0000 7f28a359e700 10 monclient(hunting): authenticate will time out at 1287.916937s 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40392/0 (socket says 172.21.15.12:40392) 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35980/0 (socket says 172.21.15.12:35980) 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 1 -- 172.21.15.12:0/4122963168 learned_addr learned my addr 172.21.15.12:0/4122963168 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:54732/0 (socket says 172.21.15.12:54732) 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): get_auth_request con 0x7f28a4133130 auth_method 0 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): get_auth_request con 0x7f28a412fc20 auth_method 0 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): get_auth_request con 0x7f28a4133500 auth_method 0 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:18.571 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient(hunting): handle_auth_done global_id 4517 payload 274 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 -- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 msgr2=0x7f28a4133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 -- 172.21.15.12:0/4122963168 >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 msgr2=0x7f28a4130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 --2- 172.21.15.12:0/4122963168 >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient: _finish_hunting 0 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 monclient: found mon.noname-e 2023-09-29T23:16:18.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 -- 172.21.15.12:0/4122963168 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4068ed0 con 0x7f28a4133130 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a2d9d700 1 --2- 172.21.15.12:0/4122963168 >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a1d9b700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a259c700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f28940098e0 tx=0x7f28940091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4bc24d3053ab75d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 1 -- 172.21.15.12:0/4122963168 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2894011ab0 con 0x7f28a4133130 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: dump: 2023-09-29T23:16:18.573 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: _finish_auth 0 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a359e700 5 monclient: authenticate success, global_id 4517 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 1 -- 172.21.15.12:0/4122963168 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2894011c30 con 0x7f28a4133130 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a359e700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:18.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a359e700 4 set_mon_vals no callback set 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 1 -- 172.21.15.12:0/4122963168 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2894011da0 con 0x7f28a4133130 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.557+0000 7f28a159a700 10 monclient: dump: 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.576 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:18.577 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 10 monclient: shutdown 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 -- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 msgr2=0x7f28a40a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:18.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f28940098e0 tx=0x7f28940091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 -- 172.21.15.12:0/4122963168 shutdown_connections 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133130 0x7f28a40a0e80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 --2- 172.21.15.12:0/4122963168 >> v2:172.21.15.12:3300/0 conn(0x7f28a412fc20 0x7f28a4130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 --2- 172.21.15.12:0/4122963168 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f28a4133500 0x7f28a4133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 -- 172.21.15.12:0/4122963168 >> 172.21.15.12:0/4122963168 conn(0x7f28a40a6490 msgr2=0x7f28a40a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 -- 172.21.15.12:0/4122963168 shutdown_connections 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 1 -- 172.21.15.12:0/4122963168 wait complete. 2023-09-29T23:16:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 10 monclient: build_initial_monmap 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.558+0000 7f28a359e700 10 monclient: monmap: 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:18.580 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 Processor -- start 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 -- start start 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient: init 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient: _add_conns ranks=[2,7,8,5,3,4,0,1,6] 2023-09-29T23:16:18.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 0x7f28a406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f28a412fc20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f28a4133130 0x7f28a406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-i con 0x7f28a4133130 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): picked mon.noname-e con 0x7f28a4133500 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a409b3e0 con 0x7f28a412fc20 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f28a409b020 con 0x7f28a4133130 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f28a411caf0 con 0x7f28a4133500 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a359e700 10 monclient(hunting): authenticate will time out at 1287.919776s 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:54742/0 (socket says 172.21.15.12:54742) 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a259c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 0x7f28a406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 1 -- 172.21.15.12:0/1381339587 learned_addr learned my addr 172.21.15.12:0/1381339587 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 10 monclient(hunting): get_auth_request con 0x7f28a4133500 auth_method 0 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a259c700 10 monclient(hunting): get_auth_request con 0x7f28a412fc20 auth_method 0 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.559+0000 7f28a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a1d9b700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f28a4133130 0x7f28a406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a1d9b700 10 monclient(hunting): get_auth_request con 0x7f28a4133130 auth_method 0 2023-09-29T23:16:18.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 -- 172.21.15.12:0/1381339587 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f28a4133130 msgr2=0x7f28a406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f28a4133130 0x7f28a406da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 -- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 msgr2=0x7f28a406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 0x7f28a406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 monclient: found mon.noname-e 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 -- 172.21.15.12:0/1381339587 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009ae0 con 0x7f28a4133500 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a259c700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 0x7f28a406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a2d9d700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f289c009fd0 tx=0x7f289c00b040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f289c018070 con 0x7f28a4133500 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: dump: 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c00f040 con 0x7f28a4133500 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 5 monclient: authenticate success, global_id 4526 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 10 monclient: _renew_subs 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2898009420 con 0x7f28a4133500 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f289c0092e0 con 0x7f28a4133500 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:18.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a0d99700 4 set_mon_vals no callback set 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28927fc700 10 monclient: dump: 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:18.589 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 10 monclient: _renew_subs 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.560+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894009420 con 0x7f28a4133500 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f289c003020 con 0x7f28a4133500 2023-09-29T23:16:18.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a359e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a404c960 con 0x7f28a4133500 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28927fc700 1 --2- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 0x7f2884038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f289c047170 con 0x7f28a4133500 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a259c700 1 --2- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 0x7f2884038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.561+0000 7f28a259c700 10 monclient: get_auth_request con 0x7f2884036780 auth_method 0 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.562+0000 7f28a259c700 1 --2- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 0x7f2884038c30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2894000c00 tx=0x7f2894017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.566+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f289c012340 con 0x7f28a4133500 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.566+0000 7f28927fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:18.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.566+0000 7f28927fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.802+0000 7f28a359e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.802+0000 7f28a359e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.802+0000 7f28a359e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.802+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28a409efd0 con 0x7f28a4133500 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.803+0000 7f28927fc700 1 -- 172.21.15.12:0/1381339587 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f289c01cbb0 con 0x7f28a4133500 2023-09-29T23:16:18.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.803+0000 7f28927fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:18.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.803+0000 7f28927fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:18.804 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:18.804 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:18.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 msgr2=0x7f2884038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 0x7f2884038c30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2894000c00 tx=0x7f2894017000 comp rx=0 tx=0).stop 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 10 monclient: shutdown 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 msgr2=0x7f28a406df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f289c009fd0 tx=0x7f289c00b040 comp rx=0 tx=0).stop 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 shutdown_connections 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f28a4133500 0x7f28a406df50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> v2:172.21.15.153:6812/83257 conn(0x7f2884036780 0x7f2884038c30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f28a412fc20 0x7f28a406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 --2- 172.21.15.12:0/1381339587 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f28a4133130 0x7f28a406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:18.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 >> 172.21.15.12:0/1381339587 conn(0x7f28a40a6490 msgr2=0x7f28a40a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:18.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 shutdown_connections 2023-09-29T23:16:18.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:18.805+0000 7f28a359e700 1 -- 172.21.15.12:0/1381339587 wait complete. 2023-09-29T23:16:21.818 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.948+0000 7f566db1c700 10 monclient: get_monmap_and_config 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.948+0000 7f566db1c700 10 monclient: build_initial_monmap 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.948+0000 7f566db1c700 10 monclient: monmap: 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:21.949628+0000 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:21.949628+0000 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:21.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.949+0000 7f566db1c700 1 Processor -- start 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.949+0000 7f566db1c700 1 -- start start 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.949+0000 7f566db1c700 10 monclient: init 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient: _add_conns ranks=[0,10,1,2,3,5,8,9,7,6,4] 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5668133130 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:21.962 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5668133c10 0x7f56680a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-f con 0x7f5668133c10 addr v2:172.21.15.153:3300/0 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5668138670 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5668068ed0 con 0x7f5668133130 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566804c960 con 0x7f5668138670 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566809b2a0 con 0x7f5668133c10 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 10 monclient(hunting): authenticate will time out at 1291.310158s 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566cb1a700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f5668133130 0x7f5668133500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:36518/0 (socket says 172.21.15.12:36518) 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566cb1a700 1 -- 172.21.15.12:0/1377107704 learned_addr learned my addr 172.21.15.12:0/1377107704 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:21.963 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 1 --2- 172.21.15.12:0/1377107704 >> v2:172.21.15.153:3300/0 conn(0x7f5668133c10 0x7f56680a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): get_auth_request con 0x7f5668133c10 auth_method 0 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928487395 0 0) 0x7f5668068ed0 con 0x7f5668133130 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): my global_id is 4770 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5654003570 con 0x7f5668133130 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.964 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141450997 0 0) 0x7f5654003570 con 0x7f5668133130 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 >> v2:172.21.15.153:3300/0 conn(0x7f5668133c10 msgr2=0x7f56680a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 --2- 172.21.15.12:0/1377107704 >> v2:172.21.15.153:3300/0 conn(0x7f5668133c10 0x7f56680a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 >> v1:172.21.15.153:6789/0 conn(0x7f5668138670 legacy=0x7f566813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: _finish_hunting 0 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56680a27a0 con 0x7f5668133130 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: _finish_auth 0 2023-09-29T23:16:21.965 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f56500018f0 con 0x7f5668133130 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 5 monclient: authenticate success, global_id 4770 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: dump: 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.966 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.967 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 >> v1:172.21.15.12:6789/0 conn(0x7f5668133130 legacy=0x7f5668133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient: _add_conns ranks=[4,5,8,3,6,2,7,0,1] 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 0x7f5654004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): picked mon.c con 0x7f56540042b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5654004da0 0x7f5654005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): picked mon.h con 0x7f5654004da0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5654009cd0 0x7f565400c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): picked mon.e con 0x7f5654009cd0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f566809b2a0 con 0x7f56540042b0 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.968 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f565400ca10 con 0x7f5654004da0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f565400cb80 con 0x7f5654009cd0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f56677fe700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.950+0000 7f566db1c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 0x7f5654004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): get_auth_request con 0x7f56540042b0 auth_method 0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5654009cd0 0x7f565400c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.969 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5654004da0 0x7f5654005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): get_auth_request con 0x7f5654009cd0 auth_method 0 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): get_auth_request con 0x7f5654004da0 auth_method 0 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.970 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient(hunting): handle_auth_done global_id 4770 payload 293 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 -- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5654009cd0 msgr2=0x7f565400c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5654009cd0 0x7f565400c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 -- 172.21.15.12:0/1377107704 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5654004da0 msgr2=0x7f5654005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5654004da0 0x7f5654005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient: _finish_hunting 0 2023-09-29T23:16:21.971 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 monclient: found mon.c 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 -- 172.21.15.12:0/1377107704 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654003570 con 0x7f56540042b0 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 10 monclient: _finish_auth 0 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f5654009cd0 0x7f565400c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566d31b700 1 -- 172.21.15.12:0/1377107704 reap_dead start 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 0x7f5654004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f565000c110 tx=0x7f5668130c30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7bd2fd37c292a172 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f565000ce80 con 0x7f56540042b0 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:21.972 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: dump: 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:21.973 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650005920 con 0x7f56540042b0 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566db1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566db1c700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 1 -- 172.21.15.12:0/1377107704 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f56500070d0 con 0x7f56540042b0 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f566db1c700 4 set_mon_vals no callback set 2023-09-29T23:16:21.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.951+0000 7f56677fe700 10 monclient: dump: 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:21.975 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 10 monclient: shutdown 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 -- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 msgr2=0x7f5654004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 0x7f5654004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f565000c110 tx=0x7f5668130c30 comp rx=0 tx=0).stop 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 -- 172.21.15.12:0/1377107704 shutdown_connections 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 --2- 172.21.15.12:0/1377107704 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f56540042b0 0x7f5654004810 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 -- 172.21.15.12:0/1377107704 >> 172.21.15.12:0/1377107704 conn(0x7f56680a6490 msgr2=0x7f56680a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 -- 172.21.15.12:0/1377107704 shutdown_connections 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 1 -- 172.21.15.12:0/1377107704 wait complete. 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 10 monclient: build_initial_monmap 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.952+0000 7f566db1c700 10 monclient: monmap: 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:21.977 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 Processor -- start 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 -- start start 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient: init 2023-09-29T23:16:21.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient: _add_conns ranks=[8,6,2,0,5,1,4,3,7] 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-e con 0x7f56540042b0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-d con 0x7f5668142390 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 0x7f566809eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): picked mon.noname-b con 0x7f5668137c60 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566809b3e0 con 0x7f5668137c60 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f566809b020 con 0x7f5668142390 2023-09-29T23:16:21.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f5668134af0 con 0x7f56540042b0 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.953+0000 7f566db1c700 10 monclient(hunting): authenticate will time out at 1291.313976s 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 0x7f566809eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:54758/0 (socket says 172.21.15.12:54758) 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 1 -- 172.21.15.12:0/1192526154 learned_addr learned my addr 172.21.15.12:0/1192526154 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): get_auth_request con 0x7f56540042b0 auth_method 0 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): get_auth_request con 0x7f5668137c60 auth_method 0 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35984/0 (socket says 172.21.15.12:35984) 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): get_auth_request con 0x7f5668142390 auth_method 0 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient(hunting): handle_auth_done global_id 4547 payload 824 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 -- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 msgr2=0x7f5668141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 -- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 msgr2=0x7f5668137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:21.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 monclient: found mon.noname-b 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 -- 172.21.15.12:0/1192526154 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668135950 con 0x7f5668137c60 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5667fff700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566d31b700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 0x7f566809eb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56680a7560 tx=0x7f5660009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5660018070 con 0x7f5668137c60 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: dump: 2023-09-29T23:16:21.983 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:21.984 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: _finish_auth 0 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660006ca0 con 0x7f5668137c60 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 5 monclient: authenticate success, global_id 4547 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 10 monclient: _renew_subs 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5668135c30 con 0x7f5668137c60 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:21.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5660006e10 con 0x7f5668137c60 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5664ff9700 10 monclient: dump: 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:21.986 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f5666ffd700 4 set_mon_vals no callback set 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 10 monclient: _renew_subs 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.954+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56680a0550 con 0x7f5668137c60 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5660006820 con 0x7f5668137c60 2023-09-29T23:16:21.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f566db1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f566db1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f566db1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5668070bc0 con 0x7f5668137c60 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.955+0000 7f5664ff9700 1 --2- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 0x7f5648038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.956+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f56600464c0 con 0x7f5668137c60 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.956+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 0x7f5648038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.956+0000 7f566cb1a700 10 monclient: get_auth_request con 0x7f5648036880 auth_method 0 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.956+0000 7f566cb1a700 1 --2- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 0x7f5648038d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5650001730 tx=0x7f5650015000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.960+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5660006ad0 con 0x7f5668137c60 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.960+0000 7f5664ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:21.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:21.960+0000 7f5664ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:22.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f566db1c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f566db1c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f566db1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56680a0740 con 0x7f5668137c60 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f5664ff9700 1 -- 172.21.15.12:0/1192526154 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5660012350 con 0x7f5668137c60 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f5664ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.198+0000 7f5664ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:22.199 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 msgr2=0x7f5648038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 0x7f5648038d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5650001730 tx=0x7f5650015000 comp rx=0 tx=0).stop 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 10 monclient: shutdown 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 msgr2=0x7f566809eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.200+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 0x7f566809eb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f56680a7560 tx=0x7f5660009040 comp rx=0 tx=0).stop 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 shutdown_connections 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> v2:172.21.15.153:6812/83257 conn(0x7f5648036880 0x7f5648038d30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:22.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5668137c60 0x7f566809eb40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f56540042b0 0x7f5668141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 --2- 172.21.15.12:0/1192526154 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5668142390 0x7f5668137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 >> 172.21.15.12:0/1192526154 conn(0x7f56680a6490 msgr2=0x7f56680a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 shutdown_connections 2023-09-29T23:16:22.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:22.201+0000 7f566db1c700 1 -- 172.21.15.12:0/1192526154 wait complete. 2023-09-29T23:16:25.213 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:25.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.344+0000 7f828238a700 10 monclient: get_monmap_and_config 2023-09-29T23:16:25.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.344+0000 7f828238a700 10 monclient: build_initial_monmap 2023-09-29T23:16:25.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.344+0000 7f828238a700 10 monclient: monmap: 2023-09-29T23:16:25.344 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:25.345278+0000 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:25.345278+0000 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:25.345 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:25.346 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 Processor -- start 2023-09-29T23:16:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 -- start start 2023-09-29T23:16:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient: init 2023-09-29T23:16:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:25.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient: _add_conns ranks=[1,10,6,0,3,8,4,7,9,5,2] 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f827c0a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f827c0a2360 0x7f827c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): picked mon.noname-f con 0x7f827c0a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): picked mon.noname-d con 0x7f827c138980 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c068ed0 con 0x7f827c0a1f90 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c09e350 con 0x7f827c0a2360 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:25.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f827c09fcf0 con 0x7f827c138980 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.345+0000 7f828238a700 10 monclient(hunting): authenticate will time out at 1294.705816s 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f827c0a2360 0x7f827c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:34250/0 (socket says 172.21.15.12:34250) 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 -- 172.21.15.12:0/1738571405 learned_addr learned my addr 172.21.15.12:0/1738571405 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): get_auth_request con 0x7f827c138980 auth_method 0 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:25.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): get_auth_request con 0x7f827c0a2360 auth_method 0 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient(hunting): handle_auth_done global_id 4695 payload 274 2023-09-29T23:16:25.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 -- 172.21.15.12:0/1738571405 >> v2:172.21.15.153:3300/0 conn(0x7f827c0a2360 msgr2=0x7f827c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 --2- 172.21.15.12:0/1738571405 >> v2:172.21.15.153:3300/0 conn(0x7f827c0a2360 0x7f827c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 -- 172.21.15.12:0/1738571405 >> v1:172.21.15.153:6789/0 conn(0x7f827c0a1f90 legacy=0x7f827c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient: _finish_hunting 0 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 monclient: found mon.noname-d 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.346+0000 7f8280927700 1 -- 172.21.15.12:0/1738571405 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c04c960 con 0x7f827c138980 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f8280927700 1 --2- 172.21.15.12:0/1738571405 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8274009f20 tx=0x7f8274009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fd4d4ca6c695733c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 1 -- 172.21.15.12:0/1738571405 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f827400e070 con 0x7f827c138980 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:25.357 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: dump: 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:25.358 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: _finish_auth 0 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 1 -- 172.21.15.12:0/1738571405 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827400be50 con 0x7f827c138980 2023-09-29T23:16:25.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 1 -- 172.21.15.12:0/1738571405 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82740092e0 con 0x7f827c138980 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f827affd700 10 monclient: dump: 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:25.360 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f828238a700 5 monclient: authenticate success, global_id 4695 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f828238a700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.347+0000 7f828238a700 4 set_mon_vals no callback set 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 10 monclient: shutdown 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 -- 172.21.15.12:0/1738571405 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 msgr2=0x7f827c13ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 --2- 172.21.15.12:0/1738571405 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8274009f20 tx=0x7f8274009840 comp rx=0 tx=0).stop 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 -- 172.21.15.12:0/1738571405 shutdown_connections 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 --2- 172.21.15.12:0/1738571405 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f827c138980 0x7f827c13ad60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 --2- 172.21.15.12:0/1738571405 >> v2:172.21.15.153:3300/0 conn(0x7f827c0a2360 0x7f827c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 -- 172.21.15.12:0/1738571405 >> 172.21.15.12:0/1738571405 conn(0x7f827c0a3b50 msgr2=0x7f827c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:25.363 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 -- 172.21.15.12:0/1738571405 shutdown_connections 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.348+0000 7f828238a700 1 -- 172.21.15.12:0/1738571405 wait complete. 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 10 monclient: build_initial_monmap 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 10 monclient: monmap: 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:25.365 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:25.366 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 1 Processor -- start 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 1 -- start start 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 10 monclient: init 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 10 monclient: _add_conns ranks=[2,1,3,5,4,0,6,7,8] 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.349+0000 7f828238a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): picked mon.noname-b con 0x7f827c0a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 0x7f827c141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): picked mon.noname-f con 0x7f827c0a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f827c138980 0x7f827c142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): picked mon.noname-g con 0x7f827c138980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c131430 con 0x7f827c0a2360 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c09e510 con 0x7f827c0a1f90 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c131150 con 0x7f827c138980 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:25.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f828238a700 10 monclient(hunting): authenticate will time out at 1294.710053s 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50602/0 (socket says 172.21.15.12:50602) 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 1 -- 172.21.15.12:0/2363229883 learned_addr learned my addr 172.21.15.12:0/2363229883 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): get_auth_request con 0x7f827c0a1f90 auth_method 0 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 0x7f827c141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): get_auth_request con 0x7f827c0a2360 auth_method 0 2023-09-29T23:16:25.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f8280927700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f827c138980 0x7f827c142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f8280927700 10 monclient(hunting): get_auth_request con 0x7f827c138980 auth_method 0 2023-09-29T23:16:25.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f8280927700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f8280927700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f8280927700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient(hunting): handle_auth_done global_id 4726 payload 824 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 -- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f827c138980 msgr2=0x7f827c142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f827c138980 0x7f827c142500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 -- 172.21.15.12:0/2363229883 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 msgr2=0x7f827c137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 monclient: found mon.noname-f 2023-09-29T23:16:25.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 -- 172.21.15.12:0/2363229883 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274009ae0 con 0x7f827c0a2360 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827bfff700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.350+0000 7f827b7fe700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 0x7f827c141fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f826400ea50 tx=0x7f82640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82640095d0 con 0x7f827c0a2360 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: dump: 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:25.372 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:25.373 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: _finish_auth 0 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826400cc70 con 0x7f827c0a2360 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 5 monclient: authenticate success, global_id 4726 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 10 monclient: _renew_subs 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c130e70 con 0x7f827c0a2360 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:25.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f826400cde0 con 0x7f827c0a2360 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f8273fff700 10 monclient: dump: 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:25.375 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f827a7fc700 4 set_mon_vals no callback set 2023-09-29T23:16:25.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 10 monclient: _renew_subs 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.351+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c1381d0 con 0x7f827c0a2360 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f828238a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f828238a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f826400c7f0 con 0x7f827c0a2360 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f828238a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c070bc0 con 0x7f827c0a2360 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f8273fff700 1 --2- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 0x7f825c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8264047710 con 0x7f827c0a2360 2023-09-29T23:16:25.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f827bfff700 1 --2- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 0x7f825c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:25.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.352+0000 7f827bfff700 10 monclient: get_auth_request con 0x7f825c036850 auth_method 0 2023-09-29T23:16:25.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.353+0000 7f827bfff700 1 --2- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 0x7f825c038d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f826c007fd0 tx=0x7f826c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:25.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.357+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f826401be10 con 0x7f827c0a2360 2023-09-29T23:16:25.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.357+0000 7f8273fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:25.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.357+0000 7f8273fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.592+0000 7f828238a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.592+0000 7f828238a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:25.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.592+0000 7f828238a700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.592+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f827c131670 con 0x7f827c0a2360 2023-09-29T23:16:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.593+0000 7f8273fff700 1 -- 172.21.15.12:0/2363229883 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f826401b3b0 con 0x7f827c0a2360 2023-09-29T23:16:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.593+0000 7f8273fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:25.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.593+0000 7f8273fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:25.594 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:25.594 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 msgr2=0x7f825c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 0x7f825c038d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f826c007fd0 tx=0x7f826c009040 comp rx=0 tx=0).stop 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 10 monclient: shutdown 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 msgr2=0x7f827c141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 0x7f827c141fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f826400ea50 tx=0x7f82640091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 shutdown_connections 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> v2:172.21.15.153:6812/83257 conn(0x7f825c036850 0x7f825c038d00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f827c138980 0x7f827c142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f827c0a1f90 0x7f827c137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 --2- 172.21.15.12:0/2363229883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f827c0a2360 0x7f827c141fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 >> 172.21.15.12:0/2363229883 conn(0x7f827c0a3b50 msgr2=0x7f827c0a60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 shutdown_connections 2023-09-29T23:16:25.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:25.595+0000 7f828238a700 1 -- 172.21.15.12:0/2363229883 wait complete. 2023-09-29T23:16:28.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:28.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.738+0000 7f7168eb0700 10 monclient: get_monmap_and_config 2023-09-29T23:16:28.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.738+0000 7f7168eb0700 10 monclient: build_initial_monmap 2023-09-29T23:16:28.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.738+0000 7f7168eb0700 10 monclient: monmap: 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:28.739408+0000 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:28.739408+0000 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:28.739 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:28.740 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 Processor -- start 2023-09-29T23:16:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 -- start start 2023-09-29T23:16:28.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient: init 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient: _add_conns ranks=[1,2,0,3,6,9,4,8,5,7,10] 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f71640a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-b con 0x7f71640a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7164138980 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7164068ed0 con 0x7f7164138980 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716404c960 con 0x7f71640a1f90 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716409e350 con 0x7f71640a2360 2023-09-29T23:16:28.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.739+0000 7f7168eb0700 10 monclient(hunting): authenticate will time out at 1298.099930s 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7162d9d700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7164138980 0x7f716413ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47456/0 (socket says 172.21.15.12:47456) 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50604/0 (socket says 172.21.15.12:50604) 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7162d9d700 1 -- 172.21.15.12:0/330120732 learned_addr learned my addr 172.21.15.12:0/330120732 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): get_auth_request con 0x7f71640a2360 auth_method 0 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:28.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient(hunting): handle_auth_done global_id 4565 payload 274 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 -- 172.21.15.12:0/330120732 >> v1:172.21.15.153:6789/0 conn(0x7f71640a1f90 legacy=0x7f716409f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 -- 172.21.15.12:0/330120732 >> v1:172.21.15.12:6789/0 conn(0x7f7164138980 legacy=0x7f716413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 1 -- 172.21.15.12:0/330120732 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438086783 0 0) 0x7f7164068ed0 con 0x7f7164138980 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 monclient: found mon.noname-b 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 -- 172.21.15.12:0/330120732 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716409da70 con 0x7f71640a2360 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7161d9b700 1 --2- 172.21.15.12:0/330120732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7154009910 tx=0x7f71540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3beaaa8667401b6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:28.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 1 -- 172.21.15.12:0/330120732 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f715400bce0 con 0x7f71640a2360 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: dump: 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:28.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: _finish_auth 0 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 1 -- 172.21.15.12:0/330120732 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715400be60 con 0x7f71640a2360 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7168eb0700 5 monclient: authenticate success, global_id 4565 2023-09-29T23:16:28.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 1 -- 172.21.15.12:0/330120732 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f71540118a0 con 0x7f71640a2360 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f716159a700 10 monclient: dump: 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:28.754 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7168eb0700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.740+0000 7f7168eb0700 4 set_mon_vals no callback set 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 10 monclient: shutdown 2023-09-29T23:16:28.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 -- 172.21.15.12:0/330120732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 msgr2=0x7f71640a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 --2- 172.21.15.12:0/330120732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7154009910 tx=0x7f71540091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 -- 172.21.15.12:0/330120732 shutdown_connections 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 --2- 172.21.15.12:0/330120732 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f71640a2360 0x7f71640a27b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 -- 172.21.15.12:0/330120732 >> 172.21.15.12:0/330120732 conn(0x7f71640a3b50 msgr2=0x7f71640a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 -- 172.21.15.12:0/330120732 shutdown_connections 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 1 -- 172.21.15.12:0/330120732 wait complete. 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 10 monclient: build_initial_monmap 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.741+0000 7f7168eb0700 10 monclient: monmap: 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:28.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:28.757 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 Processor -- start 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 -- start start 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient: init 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient: _add_conns ranks=[5,4,0,2,3,1,7,6,8] 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f71640a1f90 0x7f7164095ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-h con 0x7f71640a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-c con 0x7f71640a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164138980 0x7f716408c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:28.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): picked mon.noname-a con 0x7f7164138980 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716408f0b0 con 0x7f7164138980 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f716409ffc0 con 0x7f71640a2360 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f716408edd0 con 0x7f71640a1f90 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.742+0000 7f7168eb0700 10 monclient(hunting): authenticate will time out at 1298.102887s 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164138980 0x7f716408c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:28.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:51094/0 (socket says 172.21.15.12:51094) 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 -- 172.21.15.12:0/2766159958 learned_addr learned my addr 172.21.15.12:0/2766159958 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): get_auth_request con 0x7f7164138980 auth_method 0 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): get_auth_request con 0x7f71640a2360 auth_method 0 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f71640a1f90 0x7f7164095ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:28.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): get_auth_request con 0x7f71640a1f90 auth_method 0 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7162d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:28.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f716259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient(hunting): handle_auth_done global_id 4657 payload 824 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 -- 172.21.15.12:0/2766159958 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f71640a1f90 msgr2=0x7f7164095ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f71640a1f90 0x7f7164095ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 -- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164138980 msgr2=0x7f716408c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164138980 0x7f716408c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 monclient: found mon.noname-c 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 -- 172.21.15.12:0/2766159958 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154009420 con 0x7f71640a2360 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7161d9b700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7154013000 tx=0x7f7154017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:28.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7154006a90 con 0x7f71640a2360 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: dump: 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:28.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 5 monclient: authenticate success, global_id 4657 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71540118a0 con 0x7f71640a2360 2023-09-29T23:16:28.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 10 monclient: _renew_subs 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71640971a0 con 0x7f71640a2360 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7154011e90 con 0x7f71640a2360 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7160d99700 4 set_mon_vals no callback set 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f715a7fc700 10 monclient: dump: 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:28.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 10 monclient: _renew_subs 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.743+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716408dee0 con 0x7f71640a2360 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7154011a10 con 0x7f71640a2360 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f715a7fc700 1 --2- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 0x7f7144038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7154048ab0 con 0x7f71640a2360 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f7168eb0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f7168eb0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f7168eb0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.744+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7164070bc0 con 0x7f71640a2360 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.745+0000 7f716259c700 1 --2- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 0x7f7144038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.745+0000 7f716259c700 10 monclient: get_auth_request con 0x7f71440367b0 auth_method 0 2023-09-29T23:16:28.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.745+0000 7f716259c700 1 --2- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 0x7f7144038c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f714c003aa0 tx=0x7f714c000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:28.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.749+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7154015070 con 0x7f71640a2360 2023-09-29T23:16:28.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.749+0000 7f715a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:28.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.749+0000 7f715a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.984+0000 7f7168eb0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.984+0000 7f7168eb0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.984+0000 7f7168eb0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.984+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7164066790 con 0x7f71640a2360 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.985+0000 7f715a7fc700 1 -- 172.21.15.12:0/2766159958 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f715402d9a0 con 0x7f71640a2360 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.985+0000 7f715a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.985+0000 7f715a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:28.986 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 msgr2=0x7f7144038c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 0x7f7144038c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f714c003aa0 tx=0x7f714c000bc0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 10 monclient: shutdown 2023-09-29T23:16:28.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 msgr2=0x7f716408c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7154013000 tx=0x7f7154017000 comp rx=0 tx=0).stop 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 shutdown_connections 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> v2:172.21.15.153:6812/83257 conn(0x7f71440367b0 0x7f7144038c60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7164138980 0x7f716408c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f71640a1f90 0x7f7164095ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 --2- 172.21.15.12:0/2766159958 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f71640a2360 0x7f716408c380 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 >> 172.21.15.12:0/2766159958 conn(0x7f71640a3b50 msgr2=0x7f716409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 shutdown_connections 2023-09-29T23:16:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:28.987+0000 7f7168eb0700 1 -- 172.21.15.12:0/2766159958 wait complete. 2023-09-29T23:16:31.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:32.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.132+0000 7f7ea7a1a700 10 monclient: get_monmap_and_config 2023-09-29T23:16:32.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.132+0000 7f7ea7a1a700 10 monclient: build_initial_monmap 2023-09-29T23:16:32.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.132+0000 7f7ea7a1a700 10 monclient: monmap: 2023-09-29T23:16:32.132 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:32.132 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:32.133108+0000 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:32.133108+0000 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:32.133 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:32.134 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.132+0000 7f7ea7a1a700 1 Processor -- start 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 -- start start 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient: init 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient: _add_conns ranks=[0,3,5,2,10,7,6,4,9,1,8] 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ea006ef60 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ea0131ca0 0x7f7ea01364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-g con 0x7f7ea0131ca0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea01369e0 0x7f7ea0138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-h con 0x7f7ea01369e0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea0068ed0 con 0x7f7ea006ef60 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea009fb10 con 0x7f7ea0131ca0 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea0131a80 con 0x7f7ea01369e0 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea7a1a700 10 monclient(hunting): authenticate will time out at 1301.493684s 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea57b6700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f7ea006ef60 0x7f7ea006f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:47470/0 (socket says 172.21.15.12:47470) 2023-09-29T23:16:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea57b6700 1 -- 172.21.15.12:0/1056405793 learned_addr learned my addr 172.21.15.12:0/1056405793 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea5fb7700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea01369e0 0x7f7ea0138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ea0131ca0 0x7f7ea01364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request con 0x7f7ea01369e0 auth_method 0 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request con 0x7f7ea0131ca0 auth_method 0 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278361386 0 0) 0x7f7ea0068ed0 con 0x7f7ea006ef60 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7e93fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7e93fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.149 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7e93fff700 10 monclient(hunting): my global_id is 4797 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.133+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8c003570 con 0x7f7ea006ef60 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:32.150 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.0 v1:172.21.15.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552870664 0 0) 0x7f7e8c003570 con 0x7f7ea006ef60 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea01369e0 msgr2=0x7f7ea0138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea01369e0 0x7f7ea0138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ea0131ca0 msgr2=0x7f7ea01364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ea0131ca0 0x7f7ea01364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: _finish_hunting 0 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 monclient: found mon.noname-a-legacy 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.12:6789/0 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 --> v1:172.21.15.12:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea004c960 con 0x7f7ea006ef60 2023-09-29T23:16:32.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: _finish_auth 0 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.0 v1:172.21.15.12:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 3231398915 0 0) 0x7f7e9c0018f0 con 0x7f7ea006ef60 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea7a1a700 5 monclient: authenticate success, global_id 4797 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea7a1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: dump: 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.152 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] but i'm connected to v1:172.21.15.12:6789/0, reconnecting 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 >> v1:172.21.15.12:6789/0 conn(0x7f7ea006ef60 legacy=0x7f7ea006f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient: _add_conns ranks=[6,2,4,1,8,3,7,0,5] 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 0x7f7e8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): picked mon.d con 0x7f7e8c0042b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e8c004da0 0x7f7e8c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): picked mon.b con 0x7f7e8c004da0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:32.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7e8c009b40 0x7f7e8c00bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): picked mon.c con 0x7f7e8c009b40 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea009fb10 con 0x7f7e8c004da0 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea0131a80 con 0x7f7e8c009b40 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00c8b0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7e93fff700 10 monclient(hunting): _finish_auth 0 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea7a1a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 0x7f7e8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e8c004da0 0x7f7e8c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): get_auth_request con 0x7f7e8c0042b0 auth_method 0 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7e8c009b40 0x7f7e8c00bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request con 0x7f7e8c004da0 auth_method 0 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request con 0x7f7e8c009b40 auth_method 0 2023-09-29T23:16:32.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_done global_id 4797 payload 293 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 -- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7e8c009b40 msgr2=0x7f7e8c00bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7e8c009b40 0x7f7e8c00bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 -- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e8c004da0 msgr2=0x7f7e8c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e8c004da0 0x7f7e8c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient: _finish_hunting 0 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 monclient: found mon.d 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 1 -- 172.21.15.12:0/1056405793 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c003570 con 0x7f7e8c0042b0 2023-09-29T23:16:32.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 10 monclient: _finish_auth 0 2023-09-29T23:16:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea57b6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea5fb7700 1 -- 172.21.15.12:0/1056405793 reap_dead start 2023-09-29T23:16:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.134+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7e8c004da0 0x7f7e8c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 0x7f7e8c004810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00c190 tx=0x7f7ea0131bc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=2fbb6e9eb5c9c993 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:32.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e9c005920 con 0x7f7e8c0042b0 2023-09-29T23:16:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: dump: 2023-09-29T23:16:32.162 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:32.163 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:32.164 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:32.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c005a80 con 0x7f7e8c0042b0 2023-09-29T23:16:32.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 4 set_mon_vals no callback set 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 1 -- 172.21.15.12:0/1056405793 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e9c005bf0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:32.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7e93fff700 10 monclient: dump: 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:32.168 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 10 monclient: shutdown 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 msgr2=0x7f7e8c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 0x7f7e8c004810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00c190 tx=0x7f7ea0131bc0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/1056405793 shutdown_connections 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/1056405793 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7e8c0042b0 0x7f7e8c004810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/1056405793 >> 172.21.15.12:0/1056405793 conn(0x7f7ea00a0ea0 msgr2=0x7f7ea00a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:32.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/1056405793 shutdown_connections 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.135+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/1056405793 wait complete. 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.136+0000 7f7ea7a1a700 10 monclient: build_initial_monmap 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.136+0000 7f7ea7a1a700 10 monclient: monmap: 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:32.171 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:32.172 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:32.173 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:32.174 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.136+0000 7f7ea7a1a700 1 Processor -- start 2023-09-29T23:16:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.136+0000 7f7ea7a1a700 1 -- start start 2023-09-29T23:16:32.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.136+0000 7f7ea7a1a700 10 monclient: init 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient: _add_conns ranks=[8,5,6,4,7,3,1,0,2] 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-e con 0x7f7e8c0042b0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea012c430 0x7f7ea006aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-h con 0x7f7ea012c430 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:32.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): picked mon.noname-d con 0x7f7ea006af70 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00a01c0 con 0x7f7ea012c430 2023-09-29T23:16:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea009d5a0 con 0x7f7ea006af70 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea009e340 con 0x7f7e8c0042b0 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea7a1a700 10 monclient(hunting): authenticate will time out at 1301.497426s 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:34262/0 (socket says 172.21.15.12:34262) 2023-09-29T23:16:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:46184/0 (socket says 172.21.15.12:46184) 2023-09-29T23:16:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 1 -- 172.21.15.12:0/2217086509 learned_addr learned my addr 172.21.15.12:0/2217086509 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request con 0x7f7ea006af70 auth_method 0 2023-09-29T23:16:32.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): get_auth_request con 0x7f7e8c0042b0 auth_method 0 2023-09-29T23:16:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea012c430 0x7f7ea006aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request con 0x7f7ea012c430 auth_method 0 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:32.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea4fb5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2023-09-29T23:16:32.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 -- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 msgr2=0x7f7ea006b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 -- 172.21.15.12:0/2217086509 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea012c430 msgr2=0x7f7ea006aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea012c430 0x7f7ea006aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient: _finish_hunting 0 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 monclient: found mon.noname-e 2023-09-29T23:16:32.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea57b6700 1 -- 172.21.15.12:0/2217086509 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea009d950 con 0x7f7e8c0042b0 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.137+0000 7f7ea5fb7700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea57b6700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00c160 tx=0x7f7e9c012000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e9c021070 con 0x7f7e8c0042b0 2023-09-29T23:16:32.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: dump: 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.185 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:32.186 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:32.187 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 5 monclient: authenticate success, global_id 4544 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c01d420 con 0x7f7e8c0042b0 2023-09-29T23:16:32.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 10 monclient: _renew_subs 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea009cf70 con 0x7f7e8c0042b0 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e9c01da10 con 0x7f7e8c0042b0 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:32.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e937fe700 4 set_mon_vals no callback set 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7e917fa700 10 monclient: dump: 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.190 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:32.191 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:32.192 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 10 monclient: _renew_subs 2023-09-29T23:16:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.138+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea006c4c0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7e9c01d590 con 0x7f7e8c0042b0 2023-09-29T23:16:32.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea7a1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea7a1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea7a1a700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7e917fa700 1 --2- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 0x7f7e84038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea009d1b0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7e9c047cf0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 0x7f7e84038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.139+0000 7f7ea4fb5700 10 monclient: get_auth_request con 0x7f7e84036850 auth_method 0 2023-09-29T23:16:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.140+0000 7f7ea4fb5700 1 --2- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 0x7f7e84038d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c004180 tx=0x7f7e940091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.144+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ea009d1b0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.144+0000 7f7e917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:32.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.144+0000 7f7e917fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.383+0000 7f7ea7a1a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.383+0000 7f7ea7a1a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.383+0000 7f7ea7a1a700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.383+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ea009d1b0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.384+0000 7f7e917fa700 1 -- 172.21.15.12:0/2217086509 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7ea009d1b0 con 0x7f7e8c0042b0 2023-09-29T23:16:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.384+0000 7f7e917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:32.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.384+0000 7f7e917fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:32.385 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:32.385 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:32.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 msgr2=0x7f7e84038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 0x7f7e84038d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c004180 tx=0x7f7e940091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 10 monclient: shutdown 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 msgr2=0x7f7ea009de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00c160 tx=0x7f7e9c012000 comp rx=0 tx=0).stop 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 shutdown_connections 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> v2:172.21.15.153:6812/83257 conn(0x7f7e84036850 0x7f7e84038d00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f7ea006af70 0x7f7ea006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7e8c0042b0 0x7f7ea009de00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 --2- 172.21.15.12:0/2217086509 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f7ea012c430 0x7f7ea006aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 >> 172.21.15.12:0/2217086509 conn(0x7f7ea00a0ea0 msgr2=0x7f7ea0097270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 shutdown_connections 2023-09-29T23:16:32.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:32.386+0000 7f7ea7a1a700 1 -- 172.21.15.12:0/2217086509 wait complete. 2023-09-29T23:16:35.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.524+0000 7fe3239e6700 10 monclient: get_monmap_and_config 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.524+0000 7fe3239e6700 10 monclient: build_initial_monmap 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.525+0000 7fe3239e6700 10 monclient: monmap: 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:35.525869+0000 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:35.525869+0000 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:35.525 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:35.526 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.525+0000 7fe3239e6700 1 Processor -- start 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.525+0000 7fe3239e6700 1 -- start start 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.525+0000 7fe3239e6700 10 monclient: init 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient: _add_conns ranks=[9,3,6,2,4,10,8,0,5,7,1] 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-a con 0x7fe31c0a1f90 addr v2:172.21.15.12:3300/0 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe31c0a2360 0x7fe31c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-g con 0x7fe31c0a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:35.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-d con 0x7fe31c133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c09e350 con 0x7fe31c0a1f90 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c09ef70 con 0x7fe31c0a2360 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c09da70 con 0x7fe31c133740 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe3239e6700 10 monclient(hunting): authenticate will time out at 1304.886296s 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:48780/0 (socket says 172.21.15.12:48780) 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56392/0 (socket says 172.21.15.12:56392) 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 -- 172.21.15.12:0/4060417714 learned_addr learned my addr 172.21.15.12:0/4060417714 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): get_auth_request con 0x7fe31c133740 auth_method 0 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): get_auth_request con 0x7fe31c0a1f90 auth_method 0 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:35.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe320f81700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe31c0a2360 0x7fe31c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe320f81700 10 monclient(hunting): get_auth_request con 0x7fe31c0a2360 auth_method 0 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe320f81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe320f81700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe320f81700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient(hunting): handle_auth_done global_id 4713 payload 274 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 -- 172.21.15.12:0/4060417714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe31c0a2360 msgr2=0x7fe31c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:35.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe31c0a2360 0x7fe31c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 -- 172.21.15.12:0/4060417714 >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 msgr2=0x7fe31c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 --2- 172.21.15.12:0/4060417714 >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient: _finish_hunting 0 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 monclient: found mon.noname-d 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 -- 172.21.15.12:0/4060417714 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c068ed0 con 0x7fe31c133740 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321782700 1 --2- 172.21.15.12:0/4060417714 >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.526+0000 7fe321f83700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe314009f80 tx=0x7fe314009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c18f3f37c0c50393 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 1 -- 172.21.15.12:0/4060417714 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe31400e070 con 0x7fe31c133740 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:35.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: dump: 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:35.540 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: _finish_auth 0 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 5 monclient: authenticate success, global_id 4713 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 1 -- 172.21.15.12:0/4060417714 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31400be80 con 0x7fe31c133740 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:35.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 4 set_mon_vals no callback set 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 1 -- 172.21.15.12:0/4060417714 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3140092e0 con 0x7fe31c133740 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe313fff700 10 monclient: dump: 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:35.542 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 10 monclient: shutdown 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 -- 172.21.15.12:0/4060417714 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 msgr2=0x7fe31c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:35.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe314009f80 tx=0x7fe314009840 comp rx=0 tx=0).stop 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 -- 172.21.15.12:0/4060417714 shutdown_connections 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe31c133740 0x7fe31c133be0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 --2- 172.21.15.12:0/4060417714 >> v2:172.21.15.12:3300/0 conn(0x7fe31c0a1f90 0x7fe31c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 --2- 172.21.15.12:0/4060417714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe31c0a2360 0x7fe31c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 -- 172.21.15.12:0/4060417714 >> 172.21.15.12:0/4060417714 conn(0x7fe31c0a3b50 msgr2=0x7fe31c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 -- 172.21.15.12:0/4060417714 shutdown_connections 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.527+0000 7fe3239e6700 1 -- 172.21.15.12:0/4060417714 wait complete. 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.528+0000 7fe3239e6700 10 monclient: build_initial_monmap 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.528+0000 7fe3239e6700 10 monclient: monmap: 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:35.544 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.528+0000 7fe3239e6700 1 Processor -- start 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.528+0000 7fe3239e6700 1 -- start start 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.528+0000 7fe3239e6700 10 monclient: init 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient: _add_conns ranks=[1,4,5,8,0,2,3,6,7] 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-f con 0x7fe31c0a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-c con 0x7fe31c0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 0x7fe31c1426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): picked mon.noname-h con 0x7fe31c133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c09ffc0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c131260 con 0x7fe31c0a2360 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c130ca0 con 0x7fe31c133740 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe3239e6700 10 monclient(hunting): authenticate will time out at 1304.889342s 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe320f81700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe320f81700 1 -- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 msgr2=0x7fe31c142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe320f81700 1 -- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 msgr2=0x7fe31c142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe320f81700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:16:35.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe320f81700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:44554/0 (socket says 172.21.15.12:44554) 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 0x7fe31c1426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 -- 172.21.15.12:0/3697390748 learned_addr learned my addr 172.21.15.12:0/3697390748 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): get_auth_request con 0x7fe31c0a1f90 auth_method 0 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): get_auth_request con 0x7fe31c133740 auth_method 0 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:35.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient(hunting): handle_auth_done global_id 4744 payload 824 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 -- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 msgr2=0x7fe31c1426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 0x7fe31c1426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 -- 172.21.15.12:0/3697390748 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 msgr2=0x7fe31c142190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient: _finish_hunting 0 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 monclient: found mon.noname-f 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321782700 1 -- 172.21.15.12:0/3697390748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009420 con 0x7fe31c0a1f90 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.529+0000 7fe321f83700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 0x7fe31c1426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe321782700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe31800ec80 tx=0x7fe3180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe31800ccd0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: dump: 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:35.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:35.551 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31800ce30 con 0x7fe31c0a1f90 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 5 monclient: authenticate success, global_id 4744 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 10 monclient: _renew_subs 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe314009ae0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3180138f0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3137fe700 4 set_mon_vals no callback set 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3117fa700 10 monclient: dump: 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:35.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 10 monclient: _renew_subs 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.530+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c04c960 con 0x7fe31c0a1f90 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3239e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3239e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3239e6700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe318013470 con 0x7fe31c0a1f90 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c070bc0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3117fa700 1 --2- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 0x7fe300038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe3180479b0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe320f81700 1 --2- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 0x7fe300038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.531+0000 7fe320f81700 10 monclient: get_auth_request con 0x7fe300036880 auth_method 0 2023-09-29T23:16:35.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.532+0000 7fe320f81700 1 --2- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 0x7fe300038d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe31c12f030 tx=0x7fe30c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.536+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe31801b3b0 con 0x7fe31c0a1f90 2023-09-29T23:16:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.536+0000 7fe3117fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:35.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.536+0000 7fe3117fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.774+0000 7fe3239e6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.774+0000 7fe3239e6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.774+0000 7fe3239e6700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.774+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe31c130910 con 0x7fe31c0a1f90 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.775+0000 7fe3117fa700 1 -- 172.21.15.12:0/3697390748 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe31801ce40 con 0x7fe31c0a1f90 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.775+0000 7fe3117fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:35.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.775+0000 7fe3117fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:35.776 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:35.776 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 msgr2=0x7fe300038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 0x7fe300038d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe31c12f030 tx=0x7fe30c008040 comp rx=0 tx=0).stop 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 10 monclient: shutdown 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 msgr2=0x7fe31c132a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe31800ec80 tx=0x7fe3180091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 shutdown_connections 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> v2:172.21.15.153:6812/83257 conn(0x7fe300036880 0x7fe300038d30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe31c0a1f90 0x7fe31c132a70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fe31c133740 0x7fe31c1426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 --2- 172.21.15.12:0/3697390748 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe31c0a2360 0x7fe31c142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 >> 172.21.15.12:0/3697390748 conn(0x7fe31c0a3b50 msgr2=0x7fe31c135bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 shutdown_connections 2023-09-29T23:16:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:35.777+0000 7fe3239e6700 1 -- 172.21.15.12:0/3697390748 wait complete. 2023-09-29T23:16:38.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.921+0000 7feed7973700 10 monclient: get_monmap_and_config 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.921+0000 7feed7973700 10 monclient: build_initial_monmap 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.921+0000 7feed7973700 10 monclient: monmap: 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:38.923001+0000 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:38.923001+0000 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:38.924 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.921+0000 7feed7973700 1 Processor -- start 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 -- start start 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient: init 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient: _add_conns ranks=[4,5,1,10,0,2,9,3,6,8,7] 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.932 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): picked mon.noname-c con 0x7feed00a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): picked mon.noname-h con 0x7feed00a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7feed0133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0068ed0 con 0x7feed0133740 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7feed009e350 con 0x7feed00a1f90 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7feed009ef70 con 0x7feed00a2360 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed7973700 10 monclient(hunting): authenticate will time out at 1308.283654s 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:38.933 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:41862/0 (socket says 172.21.15.12:41862) 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 -- 172.21.15.12:0/579606462 learned_addr learned my addr 172.21.15.12:0/579606462 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): get_auth_request con 0x7feed00a1f90 auth_method 0 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient(hunting): handle_auth_done global_id 4666 payload 274 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 -- 172.21.15.12:0/579606462 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 msgr2=0x7feed00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:38.934 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 --2- 172.21.15.12:0/579606462 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 1 -- 172.21.15.12:0/579606462 >> v1:172.21.15.153:6789/0 conn(0x7feed0133740 legacy=0x7feed0133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.922+0000 7feed570f700 10 monclient: _finish_hunting 0 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed570f700 1 monclient: found mon.noname-c 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed570f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed570f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed570f700 1 -- 172.21.15.12:0/579606462 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed004c960 con 0x7feed00a1f90 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed570f700 1 --2- 172.21.15.12:0/579606462 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feecc009a10 tx=0x7feecc009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=41aa3cbd984bd13b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 1 -- 172.21.15.12:0/579606462 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feecc00bc60 con 0x7feed00a1f90 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: dump: 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:38.935 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:38.936 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: _finish_auth 0 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 1 -- 172.21.15.12:0/579606462 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc00bde0 con 0x7feed00a1f90 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 5 monclient: authenticate success, global_id 4666 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 1 -- 172.21.15.12:0/579606462 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feecc0118a0 con 0x7feed00a1f90 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:38.937 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 4 set_mon_vals no callback set 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feec7fff700 10 monclient: dump: 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:38.938 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 10 monclient: shutdown 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 -- 172.21.15.12:0/579606462 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 msgr2=0x7feed009f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 --2- 172.21.15.12:0/579606462 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feecc009a10 tx=0x7feecc009250 comp rx=0 tx=0).stop 2023-09-29T23:16:38.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 -- 172.21.15.12:0/579606462 shutdown_connections 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 --2- 172.21.15.12:0/579606462 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7feed00a1f90 0x7feed009f5e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 --2- 172.21.15.12:0/579606462 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 -- 172.21.15.12:0/579606462 >> 172.21.15.12:0/579606462 conn(0x7feed00a3b50 msgr2=0x7feed00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.923+0000 7feed7973700 1 -- 172.21.15.12:0/579606462 shutdown_connections 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 1 -- 172.21.15.12:0/579606462 wait complete. 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 10 monclient: build_initial_monmap 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 10 monclient: monmap: 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:38.940 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 1 Processor -- start 2023-09-29T23:16:38.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 1 -- start start 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.924+0000 7feed7973700 10 monclient: init 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient: _add_conns ranks=[6,5,8,1,0,2,4,7,3] 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): picked mon.noname-d con 0x7feed00a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed0132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): picked mon.noname-h con 0x7feed00a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 0x7feed012d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): picked mon.noname-e con 0x7feed0133740 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7feed0130b70 con 0x7feed00a2360 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feed0131130 con 0x7feed00a1f90 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7feed009e490 con 0x7feed0133740 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed7973700 10 monclient(hunting): authenticate will time out at 1308.286600s 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:48796/0 (socket says 172.21.15.12:48796) 2023-09-29T23:16:38.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 1 -- 172.21.15.12:0/2334830367 learned_addr learned my addr 172.21.15.12:0/2334830367 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): get_auth_request con 0x7feed00a1f90 auth_method 0 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 0x7feed012d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): get_auth_request con 0x7feed0133740 auth_method 0 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed570f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:38.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed5f10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed4f0e700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed0132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed4f0e700 10 monclient(hunting): get_auth_request con 0x7feed00a2360 auth_method 0 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed4f0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed4f0e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.925+0000 7feed4f0e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 10 monclient(hunting): handle_auth_done global_id 4722 payload 824 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 -- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 msgr2=0x7feed012d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 0x7feed012d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:38.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 -- 172.21.15.12:0/2334830367 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 msgr2=0x7feed0132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed0132940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 10 monclient: _finish_hunting 0 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 monclient: found mon.noname-d 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 -- 172.21.15.12:0/2334830367 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc0094f0 con 0x7feed00a1f90 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed5f10700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 0x7feed012d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed570f700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feecc009880 tx=0x7feecc016000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feecc026900 con 0x7feed00a1f90 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:38.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: dump: 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:38.947 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc01f070 con 0x7feed00a1f90 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 5 monclient: authenticate success, global_id 4722 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 10 monclient: _renew_subs 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feecc01a440 con 0x7feed00a1f90 2023-09-29T23:16:38.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed012fc80 con 0x7feed00a1f90 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec77fe700 4 set_mon_vals no callback set 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feec57fa700 10 monclient: dump: 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:38.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 10 monclient: _renew_subs 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:38.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.926+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed0142700 con 0x7feed00a1f90 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7feecc026480 con 0x7feed00a1f90 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed7973700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed7973700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed7973700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed01312a0 con 0x7feed00a1f90 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feec57fa700 1 --2- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 0x7feeb4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7feecc021070 con 0x7feed00a1f90 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed4f0e700 1 --2- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 0x7feeb4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.927+0000 7feed4f0e700 10 monclient: get_auth_request con 0x7feeb4036850 auth_method 0 2023-09-29T23:16:38.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.928+0000 7feed4f0e700 1 --2- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 0x7feeb4038d00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feec0006fd0 tx=0x7feec0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.932+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feecc010380 con 0x7feed00a1f90 2023-09-29T23:16:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.932+0000 7feec57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:38.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:38.932+0000 7feec57fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feed7973700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feed7973700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feed7973700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feed0070bc0 con 0x7feed00a1f90 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feec57fa700 1 -- 172.21.15.12:0/2334830367 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7feecc02b070 con 0x7feed00a1f90 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feec57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.168+0000 7feec57fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:39.170 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:39.171 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 msgr2=0x7feeb4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 0x7feeb4038d00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feec0006fd0 tx=0x7feec0008040 comp rx=0 tx=0).stop 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 10 monclient: shutdown 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 msgr2=0x7feed01421c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:39.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feecc009880 tx=0x7feecc016000 comp rx=0 tx=0).stop 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 shutdown_connections 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> v2:172.21.15.153:6812/83257 conn(0x7feeb4036850 0x7feeb4038d00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feed0133740 0x7feed012d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7feed00a1f90 0x7feed01421c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 --2- 172.21.15.12:0/2334830367 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feed00a2360 0x7feed0132940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 >> 172.21.15.12:0/2334830367 conn(0x7feed00a3b50 msgr2=0x7feed006f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 shutdown_connections 2023-09-29T23:16:39.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:39.171+0000 7feed7973700 1 -- 172.21.15.12:0/2334830367 wait complete. 2023-09-29T23:16:42.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.311+0000 7fe3c3a5f700 10 monclient: get_monmap_and_config 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.311+0000 7fe3c3a5f700 10 monclient: build_initial_monmap 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.311+0000 7fe3c3a5f700 10 monclient: monmap: 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:42.313046+0000 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:42.313046+0000 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.311+0000 7fe3c3a5f700 1 Processor -- start 2023-09-29T23:16:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 -- start start 2023-09-29T23:16:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient: init 2023-09-29T23:16:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient: _add_conns ranks=[3,0,2,8,5,4,10,1,6,9,7] 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3bc0a1f90 0x7fe3bc09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-g con 0x7fe3bc0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3bc0a2360 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 0x7fe3bc13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-b con 0x7fe3bc138980 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc068ed0 con 0x7fe3bc0a2360 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc09e350 con 0x7fe3bc138980 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc09ef70 con 0x7fe3bc0a1f90 2023-09-29T23:16:42.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c3a5f700 10 monclient(hunting): authenticate will time out at 1311.673533s 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 0x7fe3bc13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c0ffa700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe3bc0a2360 0x7fe3bc0a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:50398/0 (socket says 172.21.15.12:50398) 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c0ffa700 1 -- 172.21.15.12:0/3930818185 learned_addr learned my addr 172.21.15.12:0/3930818185 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): get_auth_request con 0x7fe3bc138980 auth_method 0 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c17fb700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3bc0a1f90 0x7fe3bc09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3b3fff700 1 -- 172.21.15.12:0/3930818185 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130380577 0 0) 0x7fe3bc068ed0 con 0x7fe3bc0a2360 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3b3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3b3fff700 10 monclient(hunting): my global_id is 4815 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3b3fff700 1 -- 172.21.15.12:0/3930818185 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a8003570 con 0x7fe3bc0a2360 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c17fb700 10 monclient(hunting): get_auth_request con 0x7fe3bc0a1f90 auth_method 0 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c17fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c17fb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c17fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.312+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_done global_id 4574 payload 274 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/3930818185 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3bc0a1f90 msgr2=0x7fe3bc09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3bc0a1f90 0x7fe3bc09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/3930818185 >> v1:172.21.15.12:6789/0 conn(0x7fe3bc0a2360 legacy=0x7fe3bc0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 10 monclient: _finish_hunting 0 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 monclient: found mon.noname-b 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/3930818185 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc04c960 con 0x7fe3bc138980 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c1ffc700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 0x7fe3bc13ad60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009e90 tx=0x7fe3b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75e9d1ed3c04f859 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 1 -- 172.21.15.12:0/3930818185 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b400e070 con 0x7fe3bc138980 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:16:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: dump: 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:42.327 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: _finish_auth 0 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 1 -- 172.21.15.12:0/3930818185 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b400bbd0 con 0x7fe3bc138980 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 5 monclient: authenticate success, global_id 4574 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 1 -- 172.21.15.12:0/3930818185 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b400bd40 con 0x7fe3bc138980 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 4 set_mon_vals no callback set 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3b3fff700 10 monclient: dump: 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:42.329 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 10 monclient: shutdown 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/3930818185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 msgr2=0x7fe3bc13ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 0x7fe3bc13ad60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009e90 tx=0x7fe3b4009780 comp rx=0 tx=0).stop 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/3930818185 shutdown_connections 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe3bc138980 0x7fe3bc13ad60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/3930818185 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3bc0a1f90 0x7fe3bc09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.313+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/3930818185 >> 172.21.15.12:0/3930818185 conn(0x7fe3bc0a3b50 msgr2=0x7fe3bc0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/3930818185 shutdown_connections 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/3930818185 wait complete. 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 10 monclient: build_initial_monmap 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 10 monclient: monmap: 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:42.332 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 1 Processor -- start 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 1 -- start start 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.314+0000 7fe3c3a5f700 10 monclient: init 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient: _add_conns ranks=[7,1,6,3,5,4,8,0,2] 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 0x7fe3bc097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-i con 0x7fe3bc0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc0a2360 0x7fe3bc1420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-f con 0x7fe3bc0a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): picked mon.noname-d con 0x7fe3bc138980 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc09e580 con 0x7fe3bc0a2360 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc08ee20 con 0x7fe3bc138980 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc08fc10 con 0x7fe3bc0a1f90 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c3a5f700 10 monclient(hunting): authenticate will time out at 1311.676569s 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:48806/0 (socket says 172.21.15.12:48806) 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/166076883 learned_addr learned my addr 172.21.15.12:0/166076883 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): get_auth_request con 0x7fe3bc138980 auth_method 0 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc0a2360 0x7fe3bc1420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): get_auth_request con 0x7fe3bc0a2360 auth_method 0 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c0ffa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 10 monclient(hunting): handle_auth_done global_id 4731 payload 824 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.315+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 msgr2=0x7fe3bc097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c17fb700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 0x7fe3bc097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 0x7fe3bc097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc0a2360 msgr2=0x7fe3bc1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc0a2360 0x7fe3bc1420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 10 monclient: _finish_hunting 0 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 monclient: found mon.noname-d 2023-09-29T23:16:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 -- 172.21.15.12:0/166076883 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b4009a20 con 0x7fe3bc138980 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c17fb700 10 monclient: get_auth_request con 0x7fe3bc0a1f90 auth_method 0 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c17fb700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 0x7fe3bc097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c1ffc700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0a0b90 tx=0x7fe3b4019000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b4012be0 con 0x7fe3bc138980 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: dump: 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: _finish_auth 0 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b4012d60 con 0x7fe3bc138980 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 5 monclient: authenticate success, global_id 4731 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b4026cf0 con 0x7fe3bc138980 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 10 monclient: _renew_subs 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc08fef0 con 0x7fe3bc138980 2023-09-29T23:16:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b37fe700 4 set_mon_vals no callback set 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3b17fa700 10 monclient: dump: 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 10 monclient: _renew_subs 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.316+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc08e0e0 con 0x7fe3bc138980 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe3b40263f0 con 0x7fe3bc138980 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c3a5f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c3a5f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c3a5f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc070bc0 con 0x7fe3bc138980 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3b17fa700 1 --2- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 0x7fe3a0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe3b4049150 con 0x7fe3bc138980 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c17fb700 1 --2- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 0x7fe3a0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.317+0000 7fe3c17fb700 10 monclient: get_auth_request con 0x7fe3a0036850 auth_method 0 2023-09-29T23:16:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.318+0000 7fe3c17fb700 1 --2- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 0x7fe3a0038d00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009990 tx=0x7fe3b8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.322+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3b4026980 con 0x7fe3bc138980 2023-09-29T23:16:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.322+0000 7fe3b17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.322+0000 7fe3b17fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3c3a5f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3c3a5f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3c3a5f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3bc066790 con 0x7fe3bc138980 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3b17fa700 1 -- 172.21.15.12:0/166076883 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe3b402e360 con 0x7fe3bc138980 2023-09-29T23:16:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3b17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.558+0000 7fe3b17fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:42.560 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:42.560 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 msgr2=0x7fe3a0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 0x7fe3a0038d00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009990 tx=0x7fe3b8008040 comp rx=0 tx=0).stop 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 10 monclient: shutdown 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 msgr2=0x7fe3bc08c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc0a0b90 tx=0x7fe3b4019000 comp rx=0 tx=0).stop 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 shutdown_connections 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> v2:172.21.15.153:6812/83257 conn(0x7fe3a0036850 0x7fe3a0038d00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe3bc138980 0x7fe3bc08c5d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3bc0a1f90 0x7fe3bc097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.562 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 --2- 172.21.15.12:0/166076883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3bc0a2360 0x7fe3bc1420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:42.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 >> 172.21.15.12:0/166076883 conn(0x7fe3bc0a3b50 msgr2=0x7fe3bc09a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:42.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 shutdown_connections 2023-09-29T23:16:42.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:42.560+0000 7fe3c3a5f700 1 -- 172.21.15.12:0/166076883 wait complete. 2023-09-29T23:16:45.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.701+0000 7f9d79b9d700 10 monclient: get_monmap_and_config 2023-09-29T23:16:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.701+0000 7f9d79b9d700 10 monclient: build_initial_monmap 2023-09-29T23:16:45.702 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.701+0000 7f9d79b9d700 10 monclient: monmap: 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:45.703500+0000 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:45.703500+0000 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:45.703 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:45.704 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 Processor -- start 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 -- start start 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient: init 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient: _add_conns ranks=[4,8,10,7,3,9,6,2,1,5,0] 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-c con 0x7f9d74133130 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9d74133500 0x7f9d74133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-e con 0x7f9d74133500 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 0x7f9d74130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.712 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-f con 0x7f9d7412fc20 addr v2:172.21.15.153:3300/0 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740a27a0 con 0x7f9d7412fc20 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7409adf0 con 0x7f9d74133130 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7409bd60 con 0x7f9d74133500 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.702+0000 7f9d79b9d700 10 monclient(hunting): authenticate will time out at 1315.063948s 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:49616/0 (socket says 172.21.15.12:49616) 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9d74133500 0x7f9d74133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 -- 172.21.15.12:0/4207615555 learned_addr learned my addr 172.21.15.12:0/4207615555 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): get_auth_request con 0x7f9d74133130 auth_method 0 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): get_auth_request con 0x7f9d74133500 auth_method 0 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d7939c700 1 --2- 172.21.15.12:0/4207615555 >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 0x7f9d74130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_done global_id 4675 payload 274 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 -- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9d74133500 msgr2=0x7f9d74133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9d74133500 0x7f9d74133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 -- 172.21.15.12:0/4207615555 >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 msgr2=0x7f9d74130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/4207615555 >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 0x7f9d74130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 monclient: found mon.noname-c 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 -- 172.21.15.12:0/4207615555 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d74068ed0 con 0x7f9d74133130 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d7939c700 10 monclient: get_auth_request con 0x7f9d7412fc20 auth_method 0 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d7939c700 1 --2- 172.21.15.12:0/4207615555 >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 0x7f9d74130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0099e0 tx=0x7f9d5c009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f402e51c04175a86 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 1 -- 172.21.15.12:0/4207615555 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d5c00bca0 con 0x7f9d74133130 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: dump: 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:45.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: _finish_auth 0 2023-09-29T23:16:45.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 1 -- 172.21.15.12:0/4207615555 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c00be20 con 0x7f9d74133130 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d79b9d700 5 monclient: authenticate success, global_id 4675 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d79b9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d79b9d700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 1 -- 172.21.15.12:0/4207615555 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d5c0118a0 con 0x7f9d74133130 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d79b9d700 4 set_mon_vals no callback set 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.703+0000 7f9d737fe700 10 monclient: dump: 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:45.718 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:45.719 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 10 monclient: shutdown 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 -- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 msgr2=0x7f9d740a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0099e0 tx=0x7f9d5c009250 comp rx=0 tx=0).stop 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 -- 172.21.15.12:0/4207615555 shutdown_connections 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/4207615555 >> v2:172.21.15.153:3300/0 conn(0x7f9d7412fc20 0x7f9d74130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9d74133130 0x7f9d740a0e80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/4207615555 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9d74133500 0x7f9d74133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 -- 172.21.15.12:0/4207615555 >> 172.21.15.12:0/4207615555 conn(0x7f9d740a6490 msgr2=0x7f9d740a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 -- 172.21.15.12:0/4207615555 shutdown_connections 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 1 -- 172.21.15.12:0/4207615555 wait complete. 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 10 monclient: build_initial_monmap 2023-09-29T23:16:45.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.704+0000 7f9d79b9d700 10 monclient: monmap: 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:45.721 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 Processor -- start 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 -- start start 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient: init 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient: _add_conns ranks=[1,3,5,0,7,4,6,2,8] 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 0x7f9d74142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-f con 0x7f9d7412fc20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 0x7f9d74142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-g con 0x7f9d74133130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): picked mon.noname-h con 0x7f9d74133500 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7409b020 con 0x7f9d7412fc20 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7411dde0 con 0x7f9d74133130 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7406f7e0 con 0x7f9d74133500 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.705+0000 7f9d79b9d700 10 monclient(hunting): authenticate will time out at 1315.066930s 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 0x7f9d74142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:33428/0 (socket says 172.21.15.12:33428) 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): get_auth_request con 0x7f9d74133130 auth_method 0 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 -- 172.21.15.12:0/1807735815 learned_addr learned my addr 172.21.15.12:0/1807735815 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 0x7f9d74142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): get_auth_request con 0x7f9d74133500 auth_method 0 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): get_auth_request con 0x7f9d7412fc20 auth_method 0 2023-09-29T23:16:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient(hunting): handle_auth_done global_id 4640 payload 824 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 -- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 msgr2=0x7f9d74142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 0x7f9d74142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 -- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 msgr2=0x7f9d74142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 0x7f9d74142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient: _finish_hunting 0 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 monclient: found mon.noname-h 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 -- 172.21.15.12:0/1807735815 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0094f0 con 0x7f9d74133500 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d73fff700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 0x7f9d74142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 0x7f9d74142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.706+0000 7f9d7939c700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c880 tx=0x7f9d6c008040 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d6c011a80 con 0x7f9d74133500 2023-09-29T23:16:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: dump: 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: _finish_auth 0 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 5 monclient: authenticate success, global_id 4640 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c011c20 con 0x7f9d74133500 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d6c011d90 con 0x7f9d74133500 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _renew_subs 2023-09-29T23:16:45.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7404c960 con 0x7f9d74133500 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d72ffd700 4 set_mon_vals no callback set 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d70ff9700 10 monclient: dump: 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _renew_subs 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7411ef40 con 0x7f9d74133500 2023-09-29T23:16:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.707+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d74070bc0 con 0x7f9d74133500 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9d6c011600 con 0x7f9d74133500 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d70ff9700 1 --2- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 0x7f9d54038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9d6c047310 con 0x7f9d74133500 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 0x7f9d54038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d78b9b700 10 monclient: get_auth_request con 0x7f9d54036850 auth_method 0 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.708+0000 7f9d78b9b700 1 --2- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 0x7f9d54038d00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0094c0 tx=0x7f9d5c017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.713+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d6c0118b0 con 0x7f9d74133500 2023-09-29T23:16:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.713+0000 7f9d70ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.713+0000 7f9d70ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:45.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.948+0000 7f9d79b9d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:45.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.948+0000 7f9d79b9d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:45.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.948+0000 7f9d79b9d700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.153:3302/0 2023-09-29T23:16:45.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.948+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9d7411ef40 con 0x7f9d74133500 2023-09-29T23:16:45.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.949+0000 7f9d70ff9700 1 -- 172.21.15.12:0/1807735815 <== mon.5 v2:172.21.15.153:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9d7411ef40 con 0x7f9d74133500 2023-09-29T23:16:45.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.949+0000 7f9d70ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:45.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.949+0000 7f9d70ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:45.950 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:45.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 msgr2=0x7f9d54038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:45.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 0x7f9d54038d00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0094c0 tx=0x7f9d5c017000 comp rx=0 tx=0).stop 2023-09-29T23:16:45.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 10 monclient: shutdown 2023-09-29T23:16:45.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 msgr2=0x7f9d74142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c880 tx=0x7f9d6c008040 comp rx=0 tx=0).stop 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 shutdown_connections 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f9d74133500 0x7f9d74142eb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7412fc20 0x7f9d74142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> v2:172.21.15.153:6812/83257 conn(0x7f9d54036850 0x7f9d54038d00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 --2- 172.21.15.12:0/1807735815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d74133130 0x7f9d74142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 >> 172.21.15.12:0/1807735815 conn(0x7f9d740a6490 msgr2=0x7f9d74130480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 shutdown_connections 2023-09-29T23:16:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:45.951+0000 7f9d79b9d700 1 -- 172.21.15.12:0/1807735815 wait complete. 2023-09-29T23:16:48.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 10 monclient: get_monmap_and_config 2023-09-29T23:16:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 10 monclient: build_initial_monmap 2023-09-29T23:16:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 10 monclient: monmap: 2023-09-29T23:16:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:49.096946+0000 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:49.096946+0000 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 1 Processor -- start 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 1 -- start start 2023-09-29T23:16:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.095+0000 7fad282c0700 10 monclient: init 2023-09-29T23:16:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient: _add_conns ranks=[1,8,0,6,2,9,4,10,3,5,7] 2023-09-29T23:16:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fad200a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 0x7fad200a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:49.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-e con 0x7fad200a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad20138980 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad20068ed0 con 0x7fad20138980 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2004c960 con 0x7fad200a1f90 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fad2009e350 con 0x7fad200a2360 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad282c0700 10 monclient(hunting): authenticate will time out at 1318.457394s 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 0x7fad200a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2685d700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fad20138980 0x7fad2013ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41708/0 (socket says 172.21.15.12:41708) 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2685d700 1 -- 172.21.15.12:0/2291347247 learned_addr learned my addr 172.21.15.12:0/2291347247 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): get_auth_request con 0x7fad200a2360 auth_method 0 2023-09-29T23:16:49.107 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2505a700 1 -- 172.21.15.12:0/2291347247 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400034306 0 0) 0x7fad20068ed0 con 0x7fad20138980 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2505a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2505a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2505a700 10 monclient(hunting): my global_id is 4824 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2505a700 1 -- 172.21.15.12:0/2291347247 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad0c003570 con 0x7fad20138980 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient(hunting): handle_auth_done global_id 4571 payload 274 2023-09-29T23:16:49.108 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 1 -- 172.21.15.12:0/2291347247 >> v1:172.21.15.153:6789/0 conn(0x7fad200a1f90 legacy=0x7fad2009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 1 -- 172.21.15.12:0/2291347247 >> v1:172.21.15.12:6789/0 conn(0x7fad20138980 legacy=0x7fad2013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 1 monclient: found mon.noname-e 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.096+0000 7fad2585b700 1 -- 172.21.15.12:0/2291347247 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2009da70 con 0x7fad200a2360 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2585b700 1 --2- 172.21.15.12:0/2291347247 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 0x7fad200a27b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad10009910 tx=0x7fad100091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b0dfe027306d82e4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 1 -- 172.21.15.12:0/2291347247 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fad1000bb00 con 0x7fad200a2360 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: dump: 2023-09-29T23:16:49.109 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:49.110 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: _finish_auth 0 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 1 -- 172.21.15.12:0/2291347247 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1000bc80 con 0x7fad200a2360 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 5 monclient: authenticate success, global_id 4571 2023-09-29T23:16:49.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 1 -- 172.21.15.12:0/2291347247 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fad1000bdf0 con 0x7fad200a2360 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 4 set_mon_vals no callback set 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad2505a700 10 monclient: dump: 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:49.112 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 10 monclient: shutdown 2023-09-29T23:16:49.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 -- 172.21.15.12:0/2291347247 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 msgr2=0x7fad200a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 --2- 172.21.15.12:0/2291347247 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 0x7fad200a27b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fad10009910 tx=0x7fad100091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 -- 172.21.15.12:0/2291347247 shutdown_connections 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 --2- 172.21.15.12:0/2291347247 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fad200a2360 0x7fad200a27b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 -- 172.21.15.12:0/2291347247 >> 172.21.15.12:0/2291347247 conn(0x7fad200a3b50 msgr2=0x7fad200a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 -- 172.21.15.12:0/2291347247 shutdown_connections 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.097+0000 7fad282c0700 1 -- 172.21.15.12:0/2291347247 wait complete. 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.098+0000 7fad282c0700 10 monclient: build_initial_monmap 2023-09-29T23:16:49.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.098+0000 7fad282c0700 10 monclient: monmap: 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:49.115 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.098+0000 7fad282c0700 1 Processor -- start 2023-09-29T23:16:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.098+0000 7fad282c0700 1 -- start start 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.098+0000 7fad282c0700 10 monclient: init 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient: _add_conns ranks=[1,0,5,6,3,7,8,4,2] 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad200a1f90 0x7fad201421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-f con 0x7fad200a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-a con 0x7fad200a2360 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fad20138980 0x7fad2008c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): picked mon.noname-h con 0x7fad20138980 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad201430c0 con 0x7fad200a2360 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2009e510 con 0x7fad200a1f90 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fad2008ec60 con 0x7fad20138980 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad282c0700 10 monclient(hunting): authenticate will time out at 1318.460324s 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45236/0 (socket says 172.21.15.12:45236) 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 -- 172.21.15.12:0/1065275489 learned_addr learned my addr 172.21.15.12:0/1065275489 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): get_auth_request con 0x7fad200a2360 auth_method 0 2023-09-29T23:16:49.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad200a1f90 0x7fad201421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fad20138980 0x7fad2008c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): get_auth_request con 0x7fad200a1f90 auth_method 0 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:49.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): get_auth_request con 0x7fad20138980 auth_method 0 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2605c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2685d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient(hunting): handle_auth_done global_id 4833 payload 824 2023-09-29T23:16:49.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 -- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fad20138980 msgr2=0x7fad2008c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fad20138980 0x7fad2008c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 -- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad200a1f90 msgr2=0x7fad201421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad200a1f90 0x7fad201421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient: _finish_hunting 0 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 monclient: found mon.noname-a 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.099+0000 7fad2585b700 1 -- 172.21.15.12:0/1065275489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10009420 con 0x7fad200a2360 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad2585b700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad2009e7d0 tx=0x7fad10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fad10006af0 con 0x7fad200a2360 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:49.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: dump: 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:49.122 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1001a430 con 0x7fad200a2360 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 5 monclient: authenticate success, global_id 4833 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 10 monclient: _renew_subs 2023-09-29T23:16:49.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2008fbc0 con 0x7fad200a2360 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fad1001aa20 con 0x7fad200a2360 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad167fc700 10 monclient: dump: 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:49.124 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:49.125 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad24859700 4 set_mon_vals no callback set 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 10 monclient: _renew_subs 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.100+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2008e020 con 0x7fad200a2360 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fad1001a5a0 con 0x7fad200a2360 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad282c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad282c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:49.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad282c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad200903a0 con 0x7fad200a2360 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad167fc700 1 --2- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 0x7fad04038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fad10048e50 con 0x7fad200a2360 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad2605c700 1 --2- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 0x7fad04038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.101+0000 7fad2605c700 10 monclient: get_auth_request con 0x7fad04036850 auth_method 0 2023-09-29T23:16:49.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.102+0000 7fad2605c700 1 --2- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 0x7fad04038d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad1c003aa0 tx=0x7fad1c000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.106+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fad200903a0 con 0x7fad200a2360 2023-09-29T23:16:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.106+0000 7fad167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:49.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.106+0000 7fad167fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.342+0000 7fad282c0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.342+0000 7fad282c0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.342+0000 7fad282c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.342+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad200903a0 con 0x7fad200a2360 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.343+0000 7fad167fc700 1 -- 172.21.15.12:0/1065275489 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fad200903a0 con 0x7fad200a2360 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.343+0000 7fad167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.343+0000 7fad167fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:49.344 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:49.345 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:49.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 msgr2=0x7fad04038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 0x7fad04038d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad1c003aa0 tx=0x7fad1c000bc0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 10 monclient: shutdown 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 msgr2=0x7fad200979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fad2009e7d0 tx=0x7fad10017000 comp rx=0 tx=0).stop 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 shutdown_connections 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> v2:172.21.15.153:6812/83257 conn(0x7fad04036850 0x7fad04038d00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fad20138980 0x7fad2008c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad200a1f90 0x7fad201421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 --2- 172.21.15.12:0/1065275489 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fad200a2360 0x7fad200979f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 >> 172.21.15.12:0/1065275489 conn(0x7fad200a3b50 msgr2=0x7fad200a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 shutdown_connections 2023-09-29T23:16:49.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:49.345+0000 7fad282c0700 1 -- 172.21.15.12:0/1065275489 wait complete. 2023-09-29T23:16:52.360 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:52.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.489+0000 7f8570cf5700 10 monclient: get_monmap_and_config 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.489+0000 7f8570cf5700 10 monclient: build_initial_monmap 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.489+0000 7f8570cf5700 10 monclient: monmap: 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:52.491516+0000 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:52.491516+0000 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:52.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:52.492 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 Processor -- start 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 -- start start 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient: init 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient: _add_conns ranks=[0,3,9,8,7,1,4,6,10,5,2] 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f856c0a1f90 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a2360 0x7f856c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-g con 0x7f856c0a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f856c138980 0x7f856c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-a con 0x7f856c138980 addr v2:172.21.15.12:3300/0 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c068ed0 con 0x7f856c0a1f90 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:52.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.490+0000 7f8570cf5700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c09e350 con 0x7f856c138980 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c09fcf0 con 0x7f856c0a2360 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 10 monclient(hunting): authenticate will time out at 1321.852016s 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856a59c700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f856c0a1f90 0x7f856c09f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:41716/0 (socket says 172.21.15.12:41716) 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856a59c700 1 -- 172.21.15.12:0/2715137675 learned_addr learned my addr 172.21.15.12:0/2715137675 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 --2- 172.21.15.12:0/2715137675 >> v2:172.21.15.12:3300/0 conn(0x7f856c138980 0x7f856c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): get_auth_request con 0x7f856c138980 auth_method 0 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8569d9b700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a2360 0x7f856c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 1 -- 172.21.15.12:0/2715137675 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346004469 0 0) 0x7f856c068ed0 con 0x7f856c0a1f90 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient(hunting): my global_id is 4851 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 1 -- 172.21.15.12:0/2715137675 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554003570 con 0x7f856c0a1f90 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8569d9b700 10 monclient(hunting): get_auth_request con 0x7f856c0a2360 auth_method 0 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8569d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:52.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8569d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8569d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient(hunting): handle_auth_done global_id 4842 payload 274 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 -- 172.21.15.12:0/2715137675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a2360 msgr2=0x7f856c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a2360 0x7f856c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 -- 172.21.15.12:0/2715137675 >> v1:172.21.15.12:6789/0 conn(0x7f856c0a1f90 legacy=0x7f856c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient: _finish_hunting 0 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 monclient: found mon.noname-a 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 -- 172.21.15.12:0/2715137675 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c04c960 con 0x7f856c138980 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856ad9d700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f856c138980 0x7f856c13ad60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8564009e60 tx=0x7f8564009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fbcf7ee9fbb7ce server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:52.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 1 -- 172.21.15.12:0/2715137675 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f856400e070 con 0x7f856c138980 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: dump: 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:52.504 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: _finish_auth 0 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 1 -- 172.21.15.12:0/2715137675 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564012430 con 0x7f856c138980 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 5 monclient: authenticate success, global_id 4842 2023-09-29T23:16:52.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 1 -- 172.21.15.12:0/2715137675 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8564012a20 con 0x7f856c138980 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f8570cf5700 4 set_mon_vals no callback set 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.491+0000 7f856959a700 10 monclient: dump: 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.506 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 10 monclient: shutdown 2023-09-29T23:16:52.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 -- 172.21.15.12:0/2715137675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f856c138980 msgr2=0x7f856c13ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f856c138980 0x7f856c13ad60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8564009e60 tx=0x7f8564009780 comp rx=0 tx=0).stop 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 -- 172.21.15.12:0/2715137675 shutdown_connections 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f856c138980 0x7f856c13ad60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 --2- 172.21.15.12:0/2715137675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a2360 0x7f856c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 -- 172.21.15.12:0/2715137675 >> 172.21.15.12:0/2715137675 conn(0x7f856c0a3b50 msgr2=0x7f856c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 -- 172.21.15.12:0/2715137675 shutdown_connections 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.492+0000 7f8570cf5700 1 -- 172.21.15.12:0/2715137675 wait complete. 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient: build_initial_monmap 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient: monmap: 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:52.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:52.509 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 Processor -- start 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 -- start start 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient: init 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient: _add_conns ranks=[3,4,8,6,7,5,0,2,1] 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a1f90 0x7f856c142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-g con 0x7f856c0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-c con 0x7f856c0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:52.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 0x7f856c08c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): picked mon.noname-e con 0x7f856c138980 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0904b0 con 0x7f856c0a1f90 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f856c08fc10 con 0x7f856c0a2360 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0901d0 con 0x7f856c138980 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8570cf5700 10 monclient(hunting): authenticate will time out at 1321.854867s 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.493+0000 7f8569d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 0x7f856c08c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:49630/0 (socket says 172.21.15.12:49630) 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 1 -- 172.21.15.12:0/978033741 learned_addr learned my addr 172.21.15.12:0/978033741 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): get_auth_request con 0x7f856c0a2360 auth_method 0 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a1f90 0x7f856c142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): get_auth_request con 0x7f856c138980 auth_method 0 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): get_auth_request con 0x7f856c0a1f90 auth_method 0 2023-09-29T23:16:52.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:52.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 -- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 msgr2=0x7f856c097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 -- 172.21.15.12:0/978033741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a1f90 msgr2=0x7f856c142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a1f90 0x7f856c142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient: _finish_hunting 0 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 monclient: found mon.noname-e 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 -- 172.21.15.12:0/978033741 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8564009a20 con 0x7f856c138980 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8569d9b700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f856ad9d700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 0x7f856c08c780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8564016000 tx=0x7f8564019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:52.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8564003310 con 0x7f856c138980 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: dump: 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:52.515 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 5 monclient: authenticate success, global_id 4580 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564012ca0 con 0x7f856c138980 2023-09-29T23:16:52.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 10 monclient: _renew_subs 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f856400bee0 con 0x7f856c138980 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c08ecb0 con 0x7f856c138980 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8568d99700 4 set_mon_vals no callback set 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f855a7fc700 10 monclient: dump: 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:52.518 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 10 monclient: _renew_subs 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.494+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c08e0d0 con 0x7f856c138980 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f856402f450 con 0x7f856c138980 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f8570cf5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f8570cf5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f8570cf5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c069ef0 con 0x7f856c138980 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f855a7fc700 1 --2- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 0x7f854c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.495+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8564048cb0 con 0x7f856c138980 2023-09-29T23:16:52.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.496+0000 7f856a59c700 1 --2- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 0x7f854c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.496+0000 7f856a59c700 10 monclient: get_auth_request con 0x7f854c036880 auth_method 0 2023-09-29T23:16:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.496+0000 7f856a59c700 1 --2- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 0x7f854c038d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f855c007fd0 tx=0x7f855c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.500+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f856402e7c0 con 0x7f856c138980 2023-09-29T23:16:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.500+0000 7f855a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:52.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.500+0000 7f855a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:52.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f8570cf5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f8570cf5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f8570cf5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f856c070bc0 con 0x7f856c138980 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f855a7fc700 1 -- 172.21.15.12:0/978033741 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f856402fe60 con 0x7f856c138980 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f855a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.734+0000 7f855a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:52.736 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 msgr2=0x7f854c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 0x7f854c038d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f855c007fd0 tx=0x7f855c009040 comp rx=0 tx=0).stop 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 10 monclient: shutdown 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 msgr2=0x7f856c08c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 0x7f856c08c780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8564016000 tx=0x7f8564019000 comp rx=0 tx=0).stop 2023-09-29T23:16:52.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 shutdown_connections 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f856c138980 0x7f856c08c780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> v2:172.21.15.153:6812/83257 conn(0x7f854c036880 0x7f854c038d30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f856c0a1f90 0x7f856c142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 --2- 172.21.15.12:0/978033741 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f856c0a2360 0x7f856c097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 >> 172.21.15.12:0/978033741 conn(0x7f856c0a3b50 msgr2=0x7f856c09a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 shutdown_connections 2023-09-29T23:16:52.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:52.737+0000 7f8570cf5700 1 -- 172.21.15.12:0/978033741 wait complete. 2023-09-29T23:16:55.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:55.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.879+0000 7f0166f13700 10 monclient: get_monmap_and_config 2023-09-29T23:16:55.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.879+0000 7f0166f13700 10 monclient: build_initial_monmap 2023-09-29T23:16:55.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.879+0000 7f0166f13700 10 monclient: monmap: 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:55.881296+0000 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:55.881296+0000 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:55.882 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 Processor -- start 2023-09-29T23:16:55.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 -- start start 2023-09-29T23:16:55.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient: init 2023-09-29T23:16:55.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient: _add_conns ranks=[1,4,5,7,8,2,6,9,3,0,10] 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f01600a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-c con 0x7f01600a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 0x7f016013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-h con 0x7f0160138980 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0160068ed0 con 0x7f01600a1f90 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f016009e350 con 0x7f01600a2360 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f016009fcf0 con 0x7f0160138980 2023-09-29T23:16:55.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f0166f13700 10 monclient(hunting): authenticate will time out at 1325.241778s 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:55910/0 (socket says 172.21.15.12:55910) 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 1 -- 172.21.15.12:0/579352573 learned_addr learned my addr 172.21.15.12:0/579352573 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 10 monclient(hunting): get_auth_request con 0x7f01600a2360 auth_method 0 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.880+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:55.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 0x7f016013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 10 monclient(hunting): get_auth_request con 0x7f0160138980 auth_method 0 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient(hunting): handle_auth_done global_id 4693 payload 274 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 -- 172.21.15.12:0/579352573 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 msgr2=0x7f016013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 0x7f016013ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 -- 172.21.15.12:0/579352573 >> v1:172.21.15.153:6789/0 conn(0x7f01600a1f90 legacy=0x7f016009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 monclient: found mon.noname-c 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:55.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 -- 172.21.15.12:0/579352573 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016004c960 con 0x7f01600a2360 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f01654b0700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 0x7f016013ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015ffff700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f01540099e0 tx=0x7f0154009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f638ed03cdc15ddb server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 1 -- 172.21.15.12:0/579352573 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f015400bc80 con 0x7f01600a2360 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: dump: 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: _finish_auth 0 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 1 -- 172.21.15.12:0/579352573 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015400be00 con 0x7f01600a2360 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f0166f13700 5 monclient: authenticate success, global_id 4693 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f0166f13700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f0166f13700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 1 -- 172.21.15.12:0/579352573 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f01540118a0 con 0x7f01600a2360 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f0166f13700 4 set_mon_vals no callback set 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.881+0000 7f015f7fe700 10 monclient: dump: 2023-09-29T23:16:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:55.898 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 10 monclient: shutdown 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 -- 172.21.15.12:0/579352573 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 msgr2=0x7f01600a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f01540099e0 tx=0x7f0154009250 comp rx=0 tx=0).stop 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 -- 172.21.15.12:0/579352573 shutdown_connections 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0160138980 0x7f016013ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 --2- 172.21.15.12:0/579352573 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f01600a2360 0x7f01600a27b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:55.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 -- 172.21.15.12:0/579352573 >> 172.21.15.12:0/579352573 conn(0x7f01600a3b50 msgr2=0x7f01600a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 -- 172.21.15.12:0/579352573 shutdown_connections 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 1 -- 172.21.15.12:0/579352573 wait complete. 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 10 monclient: build_initial_monmap 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.882+0000 7f0166f13700 10 monclient: monmap: 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:55.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 Processor -- start 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 -- start start 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient: init 2023-09-29T23:16:55.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient: _add_conns ranks=[5,6,2,8,0,3,7,4,1] 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f01600a1f90 0x7f0160142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-h con 0x7f01600a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-d con 0x7f01600a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): picked mon.noname-b con 0x7f0160138980 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016008fc60 con 0x7f0160138980 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0160090220 con 0x7f01600a1f90 2023-09-29T23:16:55.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0160090500 con 0x7f01600a2360 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f0166f13700 10 monclient(hunting): authenticate will time out at 1325.244827s 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f01654b0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f015ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f01654b0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:32788/0 (socket says 172.21.15.12:32788) 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f01654b0700 1 -- 172.21.15.12:0/56389729 learned_addr learned my addr 172.21.15.12:0/56389729 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.883+0000 7f015ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:58152/0 (socket says 172.21.15.12:58152) 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): get_auth_request con 0x7f0160138980 auth_method 0 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:55.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): get_auth_request con 0x7f01600a2360 auth_method 0 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f01600a1f90 0x7f0160142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): get_auth_request con 0x7f01600a1f90 auth_method 0 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:55.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0164caf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient(hunting): handle_auth_done global_id 4740 payload 824 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 -- 172.21.15.12:0/56389729 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f01600a1f90 msgr2=0x7f0160142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f01600a1f90 0x7f0160142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 -- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 msgr2=0x7f016008c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:55.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 monclient: found mon.noname-d 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 -- 172.21.15.12:0/56389729 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540094f0 con 0x7f01600a2360 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f01654b0700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015ffff700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0160064b10 tx=0x7f0154017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0154011c50 con 0x7f01600a2360 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: dump: 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:55.906 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:55.907 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: _finish_auth 0 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 5 monclient: authenticate success, global_id 4740 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015401f070 con 0x7f01600a2360 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0154011e10 con 0x7f01600a2360 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 10 monclient: _renew_subs 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0160142790 con 0x7f01600a2360 2023-09-29T23:16:55.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015effd700 4 set_mon_vals no callback set 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f015cff9700 10 monclient: dump: 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:55.909 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 10 monclient: _renew_subs 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.884+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016008ed00 con 0x7f01600a2360 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f015401a430 con 0x7f01600a2360 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f0166f13700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:55.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f0166f13700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f0166f13700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016009b6e0 con 0x7f01600a2360 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f015cff9700 1 --2- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 0x7f014c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0154021070 con 0x7f01600a2360 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.885+0000 7f0164caf700 1 --2- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 0x7f014c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.886+0000 7f0164caf700 10 monclient: get_auth_request con 0x7f014c036850 auth_method 0 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.886+0000 7f0164caf700 1 --2- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 0x7f014c038d00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0150003a80 tx=0x7f0150007ec0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.890+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f015401a6e0 con 0x7f01600a2360 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.890+0000 7f015cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:55.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:55.890+0000 7f015cff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.126+0000 7f0166f13700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.126+0000 7f0166f13700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.126+0000 7f0166f13700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:16:56.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.126+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f016009e490 con 0x7f01600a2360 2023-09-29T23:16:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.128+0000 7f015cff9700 1 -- 172.21.15.12:0/56389729 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f015402b030 con 0x7f01600a2360 2023-09-29T23:16:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.128+0000 7f015cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.128+0000 7f015cff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:56.129 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:56.129 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 msgr2=0x7f014c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 0x7f014c038d00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0150003a80 tx=0x7f0150007ec0 comp rx=0 tx=0).stop 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 10 monclient: shutdown 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 msgr2=0x7f0160097a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0160064b10 tx=0x7f0154017000 comp rx=0 tx=0).stop 2023-09-29T23:16:56.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 shutdown_connections 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> v2:172.21.15.153:6812/83257 conn(0x7f014c036850 0x7f014c038d00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0160138980 0x7f016008c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f01600a1f90 0x7f0160142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 --2- 172.21.15.12:0/56389729 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f01600a2360 0x7f0160097a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.131+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 >> 172.21.15.12:0/56389729 conn(0x7f01600a3b50 msgr2=0x7f016009a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.132+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 shutdown_connections 2023-09-29T23:16:56.134 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:56.132+0000 7f0166f13700 1 -- 172.21.15.12:0/56389729 wait complete. 2023-09-29T23:16:59.148 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.277+0000 7f80a525f700 10 monclient: get_monmap_and_config 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.277+0000 7f80a525f700 10 monclient: build_initial_monmap 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.277+0000 7f80a525f700 10 monclient: monmap: 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:16:59.279317+0000 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:16:59.279317+0000 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:59.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 Processor -- start 2023-09-29T23:16:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 -- start start 2023-09-29T23:16:59.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient: init 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient: _add_conns ranks=[1,6,0,4,3,9,10,7,2,8,5] 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f80a00a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 0x7f80a00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-d con 0x7f80a00a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80a0138980 addr v1:172.21.15.12:6789/0 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a0068ed0 con 0x7f80a0138980 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a004c960 con 0x7f80a00a1f90 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f80a009e350 con 0x7f80a00a2360 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f80a525f700 10 monclient(hunting): authenticate will time out at 1328.639738s 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 0x7f80a00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809f59e700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f80a0138980 0x7f80a013ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:37616/0 (socket says 172.21.15.12:37616) 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809f59e700 1 -- 172.21.15.12:0/2793137593 learned_addr learned my addr 172.21.15.12:0/2793137593 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): get_auth_request con 0x7f80a00a2360 auth_method 0 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:59.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.278+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 10 monclient(hunting): handle_auth_done global_id 4749 payload 274 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 1 -- 172.21.15.12:0/2793137593 >> v1:172.21.15.153:6789/0 conn(0x7f80a00a1f90 legacy=0x7f80a009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 1 -- 172.21.15.12:0/2793137593 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513976958 0 0) 0x7f80a0068ed0 con 0x7f80a0138980 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 1 -- 172.21.15.12:0/2793137593 >> v1:172.21.15.12:6789/0 conn(0x7f80a0138980 legacy=0x7f80a013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 10 monclient: _finish_hunting 0 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 1 monclient: found mon.noname-d 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:16:59.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 1 -- 172.21.15.12:0/2793137593 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a009da70 con 0x7f80a00a2360 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809e59c700 1 --2- 172.21.15.12:0/2793137593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 0x7f80a00a27b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8094009910 tx=0x7f80940091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ae6fef02b3ac1e57 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 1 -- 172.21.15.12:0/2793137593 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f809400bce0 con 0x7f80a00a2360 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: dump: 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:59.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: _finish_auth 0 2023-09-29T23:16:59.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 1 -- 172.21.15.12:0/2793137593 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809400be60 con 0x7f80a00a2360 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 5 monclient: authenticate success, global_id 4749 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 10 monclient: get_monmap_and_config success 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 1 -- 172.21.15.12:0/2793137593 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80940118a0 con 0x7f80a00a2360 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 4 set_mon_vals no callback set 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f809dd9b700 10 monclient: dump: 2023-09-29T23:16:59.294 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:59.295 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 10 monclient: shutdown 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 1 -- 172.21.15.12:0/2793137593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 msgr2=0x7f80a00a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.279+0000 7f80a525f700 1 --2- 172.21.15.12:0/2793137593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 0x7f80a00a27b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8094009910 tx=0x7f80940091a0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 1 -- 172.21.15.12:0/2793137593 shutdown_connections 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 1 --2- 172.21.15.12:0/2793137593 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f80a00a2360 0x7f80a00a27b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 1 -- 172.21.15.12:0/2793137593 >> 172.21.15.12:0/2793137593 conn(0x7f80a00a3b50 msgr2=0x7f80a00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 1 -- 172.21.15.12:0/2793137593 shutdown_connections 2023-09-29T23:16:59.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 1 -- 172.21.15.12:0/2793137593 wait complete. 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 10 monclient: build_initial_monmap 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.280+0000 7f80a525f700 10 monclient: monmap: 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:16:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 Processor -- start 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 -- start start 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient: init 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient: _reopen_session rank -1 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient: _add_conns ranks=[0,1,4,5,7,6,3,2,8] 2023-09-29T23:16:59.298 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-a con 0x7f80a00a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a00a2360 0x7f80a0096de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-f con 0x7f80a00a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 0x7f80a008c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): picked mon.noname-c con 0x7f80a0138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a008fd90 con 0x7f80a00a1f90 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0097320 con 0x7f80a00a2360 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00903c0 con 0x7f80a0138980 2023-09-29T23:16:59.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): _renew_subs 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f80a525f700 10 monclient(hunting): authenticate will time out at 1328.642693s 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58420/0 (socket says 172.21.15.12:58420) 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 0x7f80a008c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 1 -- 172.21.15.12:0/148501526 learned_addr learned my addr 172.21.15.12:0/148501526 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): get_auth_request con 0x7f80a00a1f90 auth_method 0 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): get_auth_request con 0x7f80a0138980 auth_method 0 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:59.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809e59c700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a00a2360 0x7f80a0096de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809e59c700 10 monclient(hunting): get_auth_request con 0x7f80a00a2360 auth_method 0 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:59.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.281+0000 7f809f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 10 monclient(hunting): handle_auth_done global_id 4869 payload 824 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 -- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 msgr2=0x7f80a008c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 0x7f80a008c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 -- 172.21.15.12:0/148501526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a00a2360 msgr2=0x7f80a0096de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a00a2360 0x7f80a0096de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 monclient: found mon.noname-a 2023-09-29T23:16:59.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 -- 172.21.15.12:0/148501526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094009420 con 0x7f80a00a1f90 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809f59e700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 0x7f80a008c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809ed9d700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f809000c5c0 tx=0x7f809000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8090001120 con 0x7f80a00a1f90 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: dump: 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.303 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:59.304 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: _finish_auth 0 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8090011a10 con 0x7f80a00a1f90 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 5 monclient: authenticate success, global_id 4869 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 10 monclient: _renew_subs 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a008ea90 con 0x7f80a00a1f90 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80900102e0 con 0x7f80a00a1f90 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:16:59.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f8086ffd700 10 monclient: dump: 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:16:59.306 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f809d59a700 4 set_mon_vals no callback set 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 10 monclient: _renew_subs 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.282+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a008eea0 con 0x7f80a00a1f90 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8090011b80 con 0x7f80a00a1f90 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f80a525f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f80a525f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:59.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f80a525f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a009fcf0 con 0x7f80a00a1f90 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f8086ffd700 1 --2- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 0x7f808c038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f80900462e0 con 0x7f80a00a1f90 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f809e59c700 1 --2- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 0x7f808c038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.283+0000 7f809e59c700 10 monclient: get_auth_request con 0x7f808c036780 auth_method 0 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.284+0000 7f809e59c700 1 --2- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 0x7f808c038c30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f80a006b460 tx=0x7f8094017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.288+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8090017620 con 0x7f80a00a1f90 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.288+0000 7f8086ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:16:59.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.288+0000 7f8086ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:16:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f80a525f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:16:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f80a525f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f80a525f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:16:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f80a0070bc0 con 0x7f80a00a1f90 2023-09-29T23:16:59.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f8086ffd700 1 -- 172.21.15.12:0/148501526 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8090017070 con 0x7f80a00a1f90 2023-09-29T23:16:59.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f8086ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:16:59.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.526+0000 7f8086ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:16:59.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:16:59.528 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:16:59.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 msgr2=0x7f808c038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 0x7f808c038c30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f80a006b460 tx=0x7f8094017000 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 10 monclient: shutdown 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 msgr2=0x7f80a0097de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f809000c5c0 tx=0x7f809000c8d0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 shutdown_connections 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> v2:172.21.15.153:6812/83257 conn(0x7f808c036780 0x7f808c038c30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f80a0138980 0x7f80a008c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f80a00a1f90 0x7f80a0097de0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 --2- 172.21.15.12:0/148501526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80a00a2360 0x7f80a0096de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:16:59.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 >> 172.21.15.12:0/148501526 conn(0x7f80a00a3b50 msgr2=0x7f80a009a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:16:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 shutdown_connections 2023-09-29T23:16:59.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:16:59.528+0000 7f80a525f700 1 -- 172.21.15.12:0/148501526 wait complete. 2023-09-29T23:17:02.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:02.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.672+0000 7f72a5992700 10 monclient: get_monmap_and_config 2023-09-29T23:17:02.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.672+0000 7f72a5992700 10 monclient: build_initial_monmap 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.672+0000 7f72a5992700 10 monclient: monmap: 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:02.673332+0000 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:02.673332+0000 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:02.673 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:02.674 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 Processor -- start 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 -- start start 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient: init 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient: _add_conns ranks=[10,5,7,6,2,3,1,4,9,0,8] 2023-09-29T23:17:02.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-f con 0x7f72a00a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-h con 0x7f72a00a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-i con 0x7f72a0133740 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a009e350 con 0x7f72a00a1f90 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f72a009ef70 con 0x7f72a00a2360 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f72a009da70 con 0x7f72a0133740 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient(hunting): authenticate will time out at 1332.033583s 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:50024/0 (socket says 172.21.15.12:50024) 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:52068/0 (socket says 172.21.15.12:52068) 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:58378/0 (socket says 172.21.15.12:58378) 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 -- 172.21.15.12:0/1773122415 learned_addr learned my addr 172.21.15.12:0/1773122415 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): get_auth_request con 0x7f72a00a1f90 auth_method 0 2023-09-29T23:17:02.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): get_auth_request con 0x7f72a0133740 auth_method 0 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): get_auth_request con 0x7f72a00a2360 auth_method 0 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient(hunting): handle_auth_done global_id 4552 payload 274 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 -- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 msgr2=0x7f72a00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 -- 172.21.15.12:0/1773122415 >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 msgr2=0x7f72a009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 --2- 172.21.15.12:0/1773122415 >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 monclient: found mon.noname-i 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 -- 172.21.15.12:0/1773122415 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0068ed0 con 0x7f72a0133740 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729effd700 1 --2- 172.21.15.12:0/1773122415 >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729e7fc700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729f7fe700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7298009f80 tx=0x7f7298009840 comp rx=0 tx=0).ready entity=mon.7 client_cookie=99846f9c2567c1c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 1 -- 172.21.15.12:0/1773122415 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f729800e070 con 0x7f72a0133740 2023-09-29T23:17:02.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: dump: 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:02.689 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:02.690 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: _finish_auth 0 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 5 monclient: authenticate success, global_id 4552 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 1 -- 172.21.15.12:0/1773122415 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729800bdd0 con 0x7f72a0133740 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 1 -- 172.21.15.12:0/1773122415 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72980092e0 con 0x7f72a0133740 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f729dffb700 10 monclient: dump: 2023-09-29T23:17:02.691 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.673+0000 7f72a5992700 4 set_mon_vals no callback set 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 10 monclient: shutdown 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 -- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 msgr2=0x7f72a0133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7298009f80 tx=0x7f7298009840 comp rx=0 tx=0).stop 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 -- 172.21.15.12:0/1773122415 shutdown_connections 2023-09-29T23:17:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 --2- 172.21.15.12:0/1773122415 >> v2:172.21.15.153:3300/0 conn(0x7f72a00a1f90 0x7f72a009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a0133740 0x7f72a0133be0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 --2- 172.21.15.12:0/1773122415 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a2360 0x7f72a00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 -- 172.21.15.12:0/1773122415 >> 172.21.15.12:0/1773122415 conn(0x7f72a00a3b50 msgr2=0x7f72a00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 -- 172.21.15.12:0/1773122415 shutdown_connections 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.674+0000 7f72a5992700 1 -- 172.21.15.12:0/1773122415 wait complete. 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient: build_initial_monmap 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient: monmap: 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:02.694 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:02.695 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 Processor -- start 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 -- start start 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient: init 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient: _add_conns ranks=[5,7,3,4,2,8,0,1,6] 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a1f90 0x7f72a009c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-h con 0x7f72a00a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:02.696 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-i con 0x7f72a00a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72a0133740 0x7f72a012d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): picked mon.noname-g con 0x7f72a0133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00a0030 con 0x7f72a0133740 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f72a012f060 con 0x7f72a00a1f90 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0130d70 con 0x7f72a00a2360 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:02.697 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.675+0000 7f72a5992700 10 monclient(hunting): authenticate will time out at 1332.036967s 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72a0133740 0x7f72a012d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a1f90 0x7f72a009c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:50032/0 (socket says 172.21.15.12:50032) 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 -- 172.21.15.12:0/1834898579 learned_addr learned my addr 172.21.15.12:0/1834898579 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): get_auth_request con 0x7f72a00a2360 auth_method 0 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): get_auth_request con 0x7f72a0133740 auth_method 0 2023-09-29T23:17:02.698 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): get_auth_request con 0x7f72a00a1f90 auth_method 0 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.699 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient(hunting): handle_auth_done global_id 4561 payload 824 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 -- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a1f90 msgr2=0x7f72a009c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a1f90 0x7f72a009c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 -- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72a0133740 msgr2=0x7f72a012d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72a0133740 0x7f72a012d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient: _finish_hunting 0 2023-09-29T23:17:02.700 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 monclient: found mon.noname-i 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 -- 172.21.15.12:0/1834898579 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290009420 con 0x7f72a00a2360 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729e7fc700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f729400ecb0 tx=0x7f72940091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f729400cda0 con 0x7f72a00a2360 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: dump: 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:02.701 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:02.702 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: _finish_auth 0 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7294013470 con 0x7f72a00a2360 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f72a5992700 5 monclient: authenticate success, global_id 4561 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f72a5992700 10 monclient: _renew_subs 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f72a5992700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7294009420 con 0x7f72a00a2360 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7294013a60 con 0x7f72a00a2360 2023-09-29T23:17:02.703 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f7286ffd700 10 monclient: dump: 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:02.704 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 10 monclient: _renew_subs 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.676+0000 7f729d7fa700 4 set_mon_vals no callback set 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7298009ae0 con 0x7f72a00a2360 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7294013bc0 con 0x7f72a00a2360 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a004c960 con 0x7f72a00a2360 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.677+0000 7f7286ffd700 1 --2- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 0x7f728c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.678+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f72940474a0 con 0x7f72a00a2360 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.678+0000 7f729effd700 1 --2- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 0x7f728c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.678+0000 7f729effd700 10 monclient: get_auth_request con 0x7f728c036880 auth_method 0 2023-09-29T23:17:02.706 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.678+0000 7f729effd700 1 --2- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 0x7f728c038d30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7290009ea0 tx=0x7f72900091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.684+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f729401bd00 con 0x7f72a00a2360 2023-09-29T23:17:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.684+0000 7f7286ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:02.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.684+0000 7f7286ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:02.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f72a5992700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:02.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f72a5992700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:02.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f72a5992700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:02.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f72a0070bc0 con 0x7f72a00a2360 2023-09-29T23:17:02.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f7286ffd700 1 -- 172.21.15.12:0/1834898579 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f729401bd00 con 0x7f72a00a2360 2023-09-29T23:17:02.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f7286ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:02.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.919+0000 7f7286ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:02.921 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:02.921 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 msgr2=0x7f728c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 0x7f728c038d30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7290009ea0 tx=0x7f72900091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 10 monclient: shutdown 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 msgr2=0x7f72a009cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f729400ecb0 tx=0x7f72940091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 shutdown_connections 2023-09-29T23:17:02.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> v2:172.21.15.153:6812/83257 conn(0x7f728c036880 0x7f728c038d30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72a0133740 0x7f72a012d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f72a00a1f90 0x7f72a009c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 --2- 172.21.15.12:0/1834898579 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f72a00a2360 0x7f72a009cc70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.921+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 >> 172.21.15.12:0/1834898579 conn(0x7f72a00a3b50 msgr2=0x7f72a006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.922+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 shutdown_connections 2023-09-29T23:17:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:02.922+0000 7f72a5992700 1 -- 172.21.15.12:0/1834898579 wait complete. 2023-09-29T23:17:05.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.065+0000 7fa4f306c700 10 monclient: get_monmap_and_config 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.065+0000 7fa4f306c700 10 monclient: build_initial_monmap 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.066+0000 7fa4f306c700 10 monclient: monmap: 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:06.066853+0000 2023-09-29T23:17:06.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:06.066853+0000 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:06.067 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:06.068 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:06.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.066+0000 7fa4f306c700 1 Processor -- start 2023-09-29T23:17:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.066+0000 7fa4f306c700 1 -- start start 2023-09-29T23:17:06.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.066+0000 7fa4f306c700 10 monclient: init 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient: _add_conns ranks=[7,2,8,9,6,1,4,10,5,0,3] 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-i con 0x7fa4ec0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-b con 0x7fa4ec0a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-e con 0x7fa4ec133740 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec09e350 con 0x7fa4ec0a2360 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec09ef70 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec09da70 con 0x7fa4ec133740 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f306c700 10 monclient(hunting): authenticate will time out at 1335.427326s 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33696/0 (socket says 172.21.15.12:33696) 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 1 -- 172.21.15.12:0/2619673421 learned_addr learned my addr 172.21.15.12:0/2619673421 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): get_auth_request con 0x7fa4ec0a2360 auth_method 0 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:58774/0 (socket says 172.21.15.12:58774) 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): get_auth_request con 0x7fa4ec133740 auth_method 0 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): get_auth_request con 0x7fa4ec0a1f90 auth_method 0 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.081 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient(hunting): handle_auth_done global_id 4589 payload 274 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 -- 172.21.15.12:0/2619673421 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 msgr2=0x7fa4ec09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 -- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 msgr2=0x7fa4ec0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient: _finish_hunting 0 2023-09-29T23:17:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 monclient: found mon.noname-e 2023-09-29T23:17:06.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:06.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:06.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4f1609700 1 -- 172.21.15.12:0/2619673421 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec068ed0 con 0x7fa4ec133740 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.067+0000 7fa4ebfff700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f1609700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009970 tx=0x7fa4e40091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a57ac979aa9a208d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 1 -- 172.21.15.12:0/2619673421 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa4e4011ab0 con 0x7fa4ec133740 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: dump: 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:06.084 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:06.085 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: _finish_auth 0 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 5 monclient: authenticate success, global_id 4589 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 1 -- 172.21.15.12:0/2619673421 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4011c70 con 0x7fa4ec133740 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 4 set_mon_vals no callback set 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 1 -- 172.21.15.12:0/2619673421 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa4e4011de0 con 0x7fa4ec133740 2023-09-29T23:17:06.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4eb7fe700 10 monclient: dump: 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.087 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:06.088 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 10 monclient: shutdown 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 -- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 msgr2=0x7fa4ec133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009970 tx=0x7fa4e40091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 -- 172.21.15.12:0/2619673421 shutdown_connections 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fa4ec133740 0x7fa4ec133bb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2619673421 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 -- 172.21.15.12:0/2619673421 >> 172.21.15.12:0/2619673421 conn(0x7fa4ec0a3b50 msgr2=0x7fa4ec0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 -- 172.21.15.12:0/2619673421 shutdown_connections 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.068+0000 7fa4f306c700 1 -- 172.21.15.12:0/2619673421 wait complete. 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.069+0000 7fa4f306c700 10 monclient: build_initial_monmap 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.069+0000 7fa4f306c700 10 monclient: monmap: 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:06.090 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.069+0000 7fa4f306c700 1 Processor -- start 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.069+0000 7fa4f306c700 1 -- start start 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.069+0000 7fa4f306c700 10 monclient: init 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient: _add_conns ranks=[7,3,1,6,8,5,2,4,0] 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-i con 0x7fa4ec0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec1328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-g con 0x7fa4ec0a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): picked mon.noname-f con 0x7fa4ec133740 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:17:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec143100 con 0x7fa4ec133740 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec12f0e0 con 0x7fa4ec0a2360 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec130df0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f306c700 10 monclient(hunting): authenticate will time out at 1335.430399s 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec1328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:37984/0 (socket says 172.21.15.12:37984) 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:37966/0 (socket says 172.21.15.12:37966) 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 -- 172.21.15.12:0/2363486481 learned_addr learned my addr 172.21.15.12:0/2363486481 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): get_auth_request con 0x7fa4ec0a1f90 auth_method 0 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): get_auth_request con 0x7fa4ec133740 auth_method 0 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): get_auth_request con 0x7fa4ec0a2360 auth_method 0 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f1609700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 -- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0a2360 msgr2=0x7fa4ec1328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec1328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 -- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 msgr2=0x7fa4ec12d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 10 monclient: _finish_hunting 0 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 1 monclient: found mon.noname-i 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.070+0000 7fa4f0e08700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f0e08700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f0e08700 1 -- 172.21.15.12:0/2363486481 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc009ae0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f1609700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f0e08700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400ece0 tx=0x7fa4d40091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d400ccd0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:17:06.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: dump: 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:06.098 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: _finish_auth 0 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d400ce70 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 5 monclient: authenticate success, global_id 4579 2023-09-29T23:17:06.099 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 10 monclient: _renew_subs 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e4009420 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d40138f0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4e8ff9700 10 monclient: dump: 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:06.100 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:06.101 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 10 monclient: _renew_subs 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4eaffd700 4 set_mon_vals no callback set 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.071+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec04c960 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4f306c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4f306c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4f306c700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa4d4013470 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec070bc0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4e8ff9700 1 --2- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 0x7fa4cc038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa4d4047850 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4ebfff700 1 --2- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 0x7fa4cc038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.072+0000 7fa4ebfff700 10 monclient: get_auth_request con 0x7fa4cc0367b0 auth_method 0 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.073+0000 7fa4ebfff700 1 --2- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 0x7fa4cc038c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0072f0 tx=0x7fa4dc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.077+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4d401b3b0 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.077+0000 7fa4e8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:06.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.077+0000 7fa4e8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:06.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.318+0000 7fa4f306c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4f306c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4f306c700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:17:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa4ec066790 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4e8ff9700 1 -- 172.21.15.12:0/2363486481 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa4d401ce40 con 0x7fa4ec0a1f90 2023-09-29T23:17:06.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4e8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:06.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.319+0000 7fa4e8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:06.320 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:06.320 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:06.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 msgr2=0x7fa4cc038c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:06.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 0x7fa4cc038c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0072f0 tx=0x7fa4dc00b040 comp rx=0 tx=0).stop 2023-09-29T23:17:06.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 10 monclient: shutdown 2023-09-29T23:17:06.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 msgr2=0x7fa4ec142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.321+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400ece0 tx=0x7fa4d40091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 shutdown_connections 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec133740 0x7fa4ec12d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fa4ec0a1f90 0x7fa4ec142200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> v2:172.21.15.153:6812/83257 conn(0x7fa4cc0367b0 0x7fa4cc038c60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 --2- 172.21.15.12:0/2363486481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0a2360 0x7fa4ec1328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 >> 172.21.15.12:0/2363486481 conn(0x7fa4ec0a3b50 msgr2=0x7fa4ec06f280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 shutdown_connections 2023-09-29T23:17:06.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:06.322+0000 7fa4f306c700 1 -- 172.21.15.12:0/2363486481 wait complete. 2023-09-29T23:17:09.334 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.466+0000 7f93a66f4700 10 monclient: get_monmap_and_config 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.466+0000 7f93a66f4700 10 monclient: build_initial_monmap 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.467+0000 7f93a66f4700 10 monclient: monmap: 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:09.467822+0000 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:09.467822+0000 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:09.467 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:09.468 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.467+0000 7f93a66f4700 1 Processor -- start 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.467+0000 7f93a66f4700 1 -- start start 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.467+0000 7f93a66f4700 10 monclient: init 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient: _add_conns ranks=[5,3,10,6,7,9,2,0,1,8,4] 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-h con 0x7f93a00a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 0x7f93a00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-g con 0x7f93a00a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 0x7f93a0133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-f con 0x7f93a0133740 addr v2:172.21.15.153:3300/0 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a009e350 con 0x7f93a0133740 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a009ef70 con 0x7f93a00a2360 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f93a009da70 con 0x7f93a00a1f90 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:09.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a66f4700 10 monclient(hunting): authenticate will time out at 1338.828362s 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 0x7f93a00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 0x7f93a0133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:36500/0 (socket says 172.21.15.12:36500) 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 1 -- 172.21.15.12:0/645326750 learned_addr learned my addr 172.21.15.12:0/645326750 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): get_auth_request con 0x7f93a00a1f90 auth_method 0 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): get_auth_request con 0x7f93a00a2360 auth_method 0 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.480 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): get_auth_request con 0x7f93a0133740 auth_method 0 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.468+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 10 monclient(hunting): handle_auth_done global_id 4694 payload 274 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 -- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 msgr2=0x7f93a00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 0x7f93a00a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 -- 172.21.15.12:0/645326750 >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 msgr2=0x7f93a0133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 --2- 172.21.15.12:0/645326750 >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 0x7f93a0133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 monclient: found mon.noname-h 2023-09-29T23:17:09.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 -- 172.21.15.12:0/645326750 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a0068ed0 con 0x7f93a00a1f90 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939f7fe700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 0x7f93a00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f93a4c91700 1 --2- 172.21.15.12:0/645326750 >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 0x7f93a0133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939ffff700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f93900099e0 tx=0x7f9390009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=909a4c01c1815e73 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 1 -- 172.21.15.12:0/645326750 <== mon.5 v2:172.21.15.153:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f939000bac0 con 0x7f93a00a1f90 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: dump: 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:09.483 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:09.484 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: _finish_auth 0 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 1 -- 172.21.15.12:0/645326750 <== mon.5 v2:172.21.15.153:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939000bc40 con 0x7f93a00a1f90 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f93a66f4700 5 monclient: authenticate success, global_id 4694 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f93a66f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 1 -- 172.21.15.12:0/645326750 <== mon.5 v2:172.21.15.153:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f939000bdb0 con 0x7f93a00a1f90 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f93a66f4700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f93a66f4700 4 set_mon_vals no callback set 2023-09-29T23:17:09.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.469+0000 7f939effd700 10 monclient: dump: 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:09.486 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 10 monclient: shutdown 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 -- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 msgr2=0x7f93a009f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f93900099e0 tx=0x7f9390009250 comp rx=0 tx=0).stop 2023-09-29T23:17:09.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 -- 172.21.15.12:0/645326750 shutdown_connections 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 --2- 172.21.15.12:0/645326750 >> v2:172.21.15.153:3300/0 conn(0x7f93a0133740 0x7f93a0133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f93a00a1f90 0x7f93a009f5e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 --2- 172.21.15.12:0/645326750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a2360 0x7f93a00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 -- 172.21.15.12:0/645326750 >> 172.21.15.12:0/645326750 conn(0x7f93a00a3b50 msgr2=0x7f93a00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 -- 172.21.15.12:0/645326750 shutdown_connections 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 1 -- 172.21.15.12:0/645326750 wait complete. 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 10 monclient: build_initial_monmap 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.470+0000 7f93a66f4700 10 monclient: monmap: 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:09.488 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:09.489 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 Processor -- start 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 -- start start 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient: init 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient: _add_conns ranks=[3,2,4,7,5,8,1,6,0] 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a1f90 0x7f93a0142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-g con 0x7f93a00a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-b con 0x7f93a00a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:09.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 0x7f93a012d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): picked mon.noname-c con 0x7f93a0133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0131190 con 0x7f93a00a2360 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a012f1a0 con 0x7f93a00a1f90 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0130bd0 con 0x7f93a0133740 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a66f4700 10 monclient(hunting): authenticate will time out at 1338.831775s 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a4c91700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 0x7f93a012d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:33712/0 (socket says 172.21.15.12:33712) 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 1 -- 172.21.15.12:0/593455290 learned_addr learned my addr 172.21.15.12:0/593455290 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 10 monclient(hunting): get_auth_request con 0x7f93a00a2360 auth_method 0 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f939f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a4c91700 10 monclient(hunting): get_auth_request con 0x7f93a0133740 auth_method 0 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a4c91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a4c91700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.471+0000 7f93a4c91700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a1f90 0x7f93a0142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a4c91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): get_auth_request con 0x7f93a00a1f90 auth_method 0 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:09.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient(hunting): handle_auth_done global_id 4601 payload 824 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 -- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 msgr2=0x7f93a012d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 0x7f93a012d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 -- 172.21.15.12:0/593455290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a1f90 msgr2=0x7f93a0142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a1f90 0x7f93a0142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 monclient: found mon.noname-b 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 -- 172.21.15.12:0/593455290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394009420 con 0x7f93a00a2360 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a4c91700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 0x7f93a012d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:09.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939f7fe700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f939400ec80 tx=0x7f93940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f939400cda0 con 0x7f93a00a2360 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: dump: 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:09.495 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: _finish_auth 0 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 5 monclient: authenticate success, global_id 4601 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 10 monclient: _renew_subs 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9398009a20 con 0x7f93a00a2360 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 10 monclient: _renew_subs 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93900094f0 con 0x7f93a00a2360 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939400c920 con 0x7f93a00a2360 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:09.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f939400ca90 con 0x7f93a00a2360 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f939e7fc700 4 set_mon_vals no callback set 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.472+0000 7f9387fff700 10 monclient: dump: 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:09.498 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f93a66f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:09.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f93a66f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f93a66f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9394005a00 con 0x7f93a00a2360 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a004c960 con 0x7f93a00a2360 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f9387fff700 1 --2- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 0x7f938c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9394047a10 con 0x7f93a00a2360 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f939ffff700 1 --2- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 0x7f938c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.473+0000 7f939ffff700 10 monclient: get_auth_request con 0x7f938c036850 auth_method 0 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.474+0000 7f939ffff700 1 --2- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 0x7f938c038d00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f93900094c0 tx=0x7f9390017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.478+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9394005cb0 con 0x7f93a00a2360 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.478+0000 7f9387fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:09.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.478+0000 7f9387fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f93a66f4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:09.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f93a66f4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f93a66f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f93a009e520 con 0x7f93a00a2360 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f9387fff700 1 -- 172.21.15.12:0/593455290 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f939401a360 con 0x7f93a00a2360 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f9387fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.713+0000 7f9387fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:09.714 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 msgr2=0x7f938c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 0x7f938c038d00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f93900094c0 tx=0x7f9390017000 comp rx=0 tx=0).stop 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 10 monclient: shutdown 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 msgr2=0x7f93a01329a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:09.716 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.715+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f939400ec80 tx=0x7f93940091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 shutdown_connections 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f93a0133740 0x7f93a012d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> v2:172.21.15.153:6812/83257 conn(0x7f938c036850 0x7f938c038d00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93a00a1f90 0x7f93a0142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 --2- 172.21.15.12:0/593455290 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f93a00a2360 0x7f93a01329a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 >> 172.21.15.12:0/593455290 conn(0x7f93a00a3b50 msgr2=0x7f93a0135c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 shutdown_connections 2023-09-29T23:17:09.717 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:09.716+0000 7f93a66f4700 1 -- 172.21.15.12:0/593455290 wait complete. 2023-09-29T23:17:12.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:12.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.859+0000 7f1b9a47f700 10 monclient: get_monmap_and_config 2023-09-29T23:17:12.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.859+0000 7f1b9a47f700 10 monclient: build_initial_monmap 2023-09-29T23:17:12.859 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.859+0000 7f1b9a47f700 10 monclient: monmap: 2023-09-29T23:17:12.859 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:12.860350+0000 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:12.860350+0000 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:12.860 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:12.861 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 Processor -- start 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 -- start start 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient: init 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient: _add_conns ranks=[0,9,8,2,3,7,1,5,6,4,10] 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b9406ef60 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b94131ca0 0x7f1b941364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-a con 0x7f1b94131ca0 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:12.870 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 0x7f1b94138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-e con 0x7f1b941369e0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b94068ed0 con 0x7f1b9406ef60 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9409fb10 con 0x7f1b94131ca0 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1b94131a80 con 0x7f1b941369e0 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b9a47f700 10 monclient(hunting): authenticate will time out at 1342.220821s 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b937fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b94131ca0 0x7f1b941364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b98a1c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 0x7f1b94138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.871 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b93fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f1b9406ef60 0x7f1b9406f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:48356/0 (socket says 172.21.15.12:48356) 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b937fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f1b94131ca0 0x7f1b941364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40488/0 (socket says 172.21.15.12:40488) 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b93fff700 1 -- 172.21.15.12:0/923795176 learned_addr learned my addr 172.21.15.12:0/923795176 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.860+0000 7f1b937fe700 10 monclient(hunting): get_auth_request con 0x7f1b94131ca0 auth_method 0 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): get_auth_request con 0x7f1b941369e0 auth_method 0 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:12.872 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient(hunting): handle_auth_done global_id 4878 payload 274 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 -- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 msgr2=0x7f1b94138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 0x7f1b94138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 -- 172.21.15.12:0/923795176 >> v1:172.21.15.12:6789/0 conn(0x7f1b9406ef60 legacy=0x7f1b9406f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 monclient: found mon.noname-a 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:12.873 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 -- 172.21.15.12:0/923795176 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9404c960 con 0x7f1b94131ca0 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 1 -- 172.21.15.12:0/923795176 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086323726 0 0) 0x7f1b94068ed0 con 0x7f1b9406ef60 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b98a1c700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 0x7f1b94138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b937fe700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b94131ca0 0x7f1b941364a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1b880098e0 tx=0x7f1b880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8647d633945c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 1 -- 172.21.15.12:0/923795176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b8800bac0 con 0x7f1b94131ca0 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: dump: 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.874 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:12.875 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: _finish_auth 0 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b9a47f700 5 monclient: authenticate success, global_id 4878 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 1 -- 172.21.15.12:0/923795176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8800bc40 con 0x7f1b94131ca0 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b9a47f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b9a47f700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b9a47f700 4 set_mon_vals no callback set 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 1 -- 172.21.15.12:0/923795176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b8800bdb0 con 0x7f1b94131ca0 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:12.876 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.861+0000 7f1b92ffd700 10 monclient: dump: 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:12.877 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 10 monclient: shutdown 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 -- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b94131ca0 msgr2=0x7f1b941364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b94131ca0 0x7f1b941364a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1b880098e0 tx=0x7f1b880091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 -- 172.21.15.12:0/923795176 shutdown_connections 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1b941369e0 0x7f1b94138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/923795176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b94131ca0 0x7f1b941364a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.878 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 -- 172.21.15.12:0/923795176 >> 172.21.15.12:0/923795176 conn(0x7f1b940a0ea0 msgr2=0x7f1b940a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 -- 172.21.15.12:0/923795176 shutdown_connections 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 1 -- 172.21.15.12:0/923795176 wait complete. 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 10 monclient: build_initial_monmap 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.862+0000 7f1b9a47f700 10 monclient: monmap: 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:12.879 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 Processor -- start 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 -- start start 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient: init 2023-09-29T23:17:12.880 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient: _add_conns ranks=[4,2,5,1,8,7,6,0,3] 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-c con 0x7f1b9406ef60 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 0x7f1b9412c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-b con 0x7f1b94131ca0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1b941369e0 0x7f1b9406a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): picked mon.noname-h con 0x7f1b941369e0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9409e180 con 0x7f1b94131ca0 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9409d550 con 0x7f1b9406ef60 2023-09-29T23:17:12.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9409bdc0 con 0x7f1b941369e0 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b9a47f700 10 monclient(hunting): authenticate will time out at 1342.223910s 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.863+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 0x7f1b9412c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:41314/0 (socket says 172.21.15.12:41314) 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 -- 172.21.15.12:0/2774632373 learned_addr learned my addr 172.21.15.12:0/2774632373 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): get_auth_request con 0x7f1b9406ef60 auth_method 0 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:12.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): get_auth_request con 0x7f1b94131ca0 auth_method 0 2023-09-29T23:17:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:12.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b98a1c700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1b941369e0 0x7f1b9406a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b98a1c700 10 monclient(hunting): get_auth_request con 0x7f1b941369e0 auth_method 0 2023-09-29T23:17:12.885 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b98a1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b98a1c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b98a1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient(hunting): handle_auth_done global_id 4720 payload 824 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 -- 172.21.15.12:0/2774632373 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1b941369e0 msgr2=0x7f1b9406a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1b941369e0 0x7f1b9406a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 -- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 msgr2=0x7f1b9412c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 0x7f1b9412c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 monclient: found mon.noname-c 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:17:12.886 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 -- 172.21.15.12:0/2774632373 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c009a20 con 0x7f1b9406ef60 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b937fe700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 0x7f1b9412c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b93fff700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c840 tx=0x7f1b84007d90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b84007550 con 0x7f1b9406ef60 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: dump: 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:12.887 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: _finish_auth 0 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:12.888 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 5 monclient: authenticate success, global_id 4720 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84007710 con 0x7f1b9406ef60 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1b84005c00 con 0x7f1b9406ef60 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b927fc700 4 set_mon_vals no callback set 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b7bfff700 10 monclient: dump: 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:12.889 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:12.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 10 monclient: _renew_subs 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b88009420 con 0x7f1b9406ef60 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 10 monclient: _renew_subs 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.864+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b940a01c0 con 0x7f1b9406ef60 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1b84007070 con 0x7f1b9406ef60 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b9a47f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b9a47f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b9a47f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b94070bc0 con 0x7f1b9406ef60 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b7bfff700 1 --2- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 0x7f1b80038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:12.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.865+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1b84046480 con 0x7f1b9406ef60 2023-09-29T23:17:12.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.866+0000 7f1b937fe700 1 --2- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 0x7f1b80038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:12.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.866+0000 7f1b937fe700 10 monclient: get_auth_request con 0x7f1b80036780 auth_method 0 2023-09-29T23:17:12.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.866+0000 7f1b937fe700 1 --2- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 0x7f1b80038c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1b940a06c0 tx=0x7f1b88017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:12.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.870+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b84007320 con 0x7f1b9406ef60 2023-09-29T23:17:12.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.870+0000 7f1b7bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:12.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:12.870+0000 7f1b7bfff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.102+0000 7f1b9a47f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.102+0000 7f1b9a47f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:13.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.102+0000 7f1b9a47f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.102+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1b9409d120 con 0x7f1b9406ef60 2023-09-29T23:17:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.103+0000 7f1b7bfff700 1 -- 172.21.15.12:0/2774632373 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1b84017020 con 0x7f1b9406ef60 2023-09-29T23:17:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.103+0000 7f1b7bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:13.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.103+0000 7f1b7bfff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:13.104 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:13.104 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 msgr2=0x7f1b80038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 0x7f1b80038c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1b940a06c0 tx=0x7f1b88017000 comp rx=0 tx=0).stop 2023-09-29T23:17:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 10 monclient: shutdown 2023-09-29T23:17:13.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 msgr2=0x7f1b9409dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c840 tx=0x7f1b84007d90 comp rx=0 tx=0).stop 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 shutdown_connections 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1b941369e0 0x7f1b9406a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1b9406ef60 0x7f1b9409dc40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> v2:172.21.15.153:6812/83257 conn(0x7f1b80036780 0x7f1b80038c30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 --2- 172.21.15.12:0/2774632373 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b94131ca0 0x7f1b9412c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 >> 172.21.15.12:0/2774632373 conn(0x7f1b940a0ea0 msgr2=0x7f1b94097350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 shutdown_connections 2023-09-29T23:17:13.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:13.105+0000 7f1b9a47f700 1 -- 172.21.15.12:0/2774632373 wait complete. 2023-09-29T23:17:16.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.250+0000 7f58dfe34700 10 monclient: get_monmap_and_config 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.250+0000 7f58dfe34700 10 monclient: build_initial_monmap 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.251+0000 7f58dfe34700 10 monclient: monmap: 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:16.251857+0000 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:16.251857+0000 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:16.251 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:16.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.251+0000 7f58dfe34700 1 Processor -- start 2023-09-29T23:17:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.251+0000 7f58dfe34700 1 -- start start 2023-09-29T23:17:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.251+0000 7f58dfe34700 10 monclient: init 2023-09-29T23:17:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:16.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient: _add_conns ranks=[8,0,9,3,1,4,5,10,2,6,7] 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 0x7f58d809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-e con 0x7f58d80a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58d80a2360 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58d8138980 0x7f58d813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-a con 0x7f58d8138980 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d8068ed0 con 0x7f58d80a2360 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d809e350 con 0x7f58d8138980 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f58d809ef70 con 0x7f58d80a1f90 2023-09-29T23:17:16.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dfe34700 10 monclient(hunting): authenticate will time out at 1345.612305s 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f58d8138980 0x7f58d813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 0x7f58d809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dd3cf700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f58d80a2360 0x7f58d80a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56178/0 (socket says 172.21.15.12:56178) 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dd3cf700 1 -- 172.21.15.12:0/2919020642 learned_addr learned my addr 172.21.15.12:0/2919020642 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): get_auth_request con 0x7f58d80a1f90 auth_method 0 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): get_auth_request con 0x7f58d8138980 auth_method 0 2023-09-29T23:17:16.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient(hunting): handle_auth_done global_id 4896 payload 274 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 -- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 msgr2=0x7f58d809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:16.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 0x7f58d809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 -- 172.21.15.12:0/2919020642 >> v1:172.21.15.12:6789/0 conn(0x7f58d80a2360 legacy=0x7f58d80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient: _finish_hunting 0 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 monclient: found mon.noname-a 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58de3d1700 1 -- 172.21.15.12:0/2919020642 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d804c960 con 0x7f58d8138980 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dcbce700 1 -- 172.21.15.12:0/2919020642 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483223408 0 0) 0x7f58d8068ed0 con 0x7f58d80a2360 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58dcbce700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.252+0000 7f58ddbd0700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 0x7f58d809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58de3d1700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d8138980 0x7f58d813ad60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009e90 tx=0x7f58d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c19d5b4d7d4ca64 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:16.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 1 -- 172.21.15.12:0/2919020642 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58d000e070 con 0x7f58d8138980 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: dump: 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:16.266 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: _finish_auth 0 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 5 monclient: authenticate success, global_id 4896 2023-09-29T23:17:16.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 1 -- 172.21.15.12:0/2919020642 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d000bbf0 con 0x7f58d8138980 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 1 -- 172.21.15.12:0/2919020642 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58d000bd60 con 0x7f58d8138980 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 4 set_mon_vals no callback set 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dcbce700 10 monclient: dump: 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.268 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:16.269 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 10 monclient: shutdown 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 -- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d8138980 msgr2=0x7f58d813ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d8138980 0x7f58d813ad60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009e90 tx=0x7f58d0009780 comp rx=0 tx=0).stop 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 -- 172.21.15.12:0/2919020642 shutdown_connections 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58d8138980 0x7f58d813ad60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2919020642 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d80a1f90 0x7f58d809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 -- 172.21.15.12:0/2919020642 >> 172.21.15.12:0/2919020642 conn(0x7f58d80a3b50 msgr2=0x7f58d80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 -- 172.21.15.12:0/2919020642 shutdown_connections 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.253+0000 7f58dfe34700 1 -- 172.21.15.12:0/2919020642 wait complete. 2023-09-29T23:17:16.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.254+0000 7f58dfe34700 10 monclient: build_initial_monmap 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.254+0000 7f58dfe34700 10 monclient: monmap: 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:16.271 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.254+0000 7f58dfe34700 1 Processor -- start 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.254+0000 7f58dfe34700 1 -- start start 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.254+0000 7f58dfe34700 10 monclient: init 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient: _add_conns ranks=[1,4,8,5,2,7,6,0,3] 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 0x7f58d8141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-f con 0x7f58d80a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:17:16.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f58d80a2360 0x7f58d8137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-c con 0x7f58d80a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): picked mon.noname-e con 0x7f58d8138980 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d8130ae0 con 0x7f58d80a1f90 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f58d809e540 con 0x7f58d80a2360 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f58d809fff0 con 0x7f58d8138980 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dfe34700 10 monclient(hunting): authenticate will time out at 1345.615474s 2023-09-29T23:17:16.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:48510/0 (socket says 172.21.15.12:48510) 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 -- 172.21.15.12:0/2916890333 learned_addr learned my addr 172.21.15.12:0/2916890333 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): get_auth_request con 0x7f58d8138980 auth_method 0 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f58d80a2360 0x7f58d8137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): get_auth_request con 0x7f58d80a2360 auth_method 0 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:16.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 0x7f58d8141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58dd3cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 10 monclient(hunting): get_auth_request con 0x7f58d80a1f90 auth_method 0 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient(hunting): handle_auth_done global_id 4616 payload 824 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 -- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f58d80a2360 msgr2=0x7f58d8137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f58d80a2360 0x7f58d8137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 -- 172.21.15.12:0/2916890333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 msgr2=0x7f58d8141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 0x7f58d8141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient: _finish_hunting 0 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 monclient: found mon.noname-e 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58de3d1700 1 -- 172.21.15.12:0/2916890333 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4009420 con 0x7f58d8138980 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.255+0000 7f58ddbd0700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 0x7f58d8141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58de3d1700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58d0013000 tx=0x7f58d0019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:16.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58d00070c0 con 0x7f58d8138980 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: dump: 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:16.277 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: _finish_auth 0 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0012c00 con 0x7f58d8138980 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 5 monclient: authenticate success, global_id 4616 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:16.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f58d0012d70 con 0x7f58d8138980 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 10 monclient: _renew_subs 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cffff700 4 set_mon_vals no callback set 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d0009a20 con 0x7f58d8138980 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58cdffb700 10 monclient: dump: 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:16.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:16.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 10 monclient: _renew_subs 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.256+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d8130240 con 0x7f58d8138980 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f58d00263f0 con 0x7f58d8138980 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58dfe34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58dfe34700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58cdffb700 1 --2- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 0x7f58bc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58dfe34700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d8132200 con 0x7f58d8138980 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f58d0048c80 con 0x7f58d8138980 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58ddbd0700 1 --2- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 0x7f58bc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.257+0000 7f58ddbd0700 10 monclient: get_auth_request con 0x7f58bc036880 auth_method 0 2023-09-29T23:17:16.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.258+0000 7f58ddbd0700 1 --2- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 0x7f58bc038d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f58d40093f0 tx=0x7f58d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.262+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58d8132200 con 0x7f58d8138980 2023-09-29T23:17:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.262+0000 7f58cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:16.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.262+0000 7f58cdffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:16.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.498+0000 7f58dfe34700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.498+0000 7f58dfe34700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.498+0000 7f58dfe34700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.498+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58d8132200 con 0x7f58d8138980 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.498+0000 7f58cdffb700 1 -- 172.21.15.12:0/2916890333 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f58d8132200 con 0x7f58d8138980 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.499+0000 7f58cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.499+0000 7f58cdffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:16.499 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 msgr2=0x7f58bc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 0x7f58bc038d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f58d40093f0 tx=0x7f58d4008040 comp rx=0 tx=0).stop 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 10 monclient: shutdown 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 msgr2=0x7f58d812e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58d0013000 tx=0x7f58d0019000 comp rx=0 tx=0).stop 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 shutdown_connections 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> v2:172.21.15.153:6812/83257 conn(0x7f58bc036880 0x7f58bc038d30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f58d8138980 0x7f58d812e670 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58d80a1f90 0x7f58d8141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 --2- 172.21.15.12:0/2916890333 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f58d80a2360 0x7f58d8137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:16.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.500+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 >> 172.21.15.12:0/2916890333 conn(0x7f58d80a3b50 msgr2=0x7f58d80a4390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:16.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.501+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 shutdown_connections 2023-09-29T23:17:16.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:16.501+0000 7f58dfe34700 1 -- 172.21.15.12:0/2916890333 wait complete. 2023-09-29T23:17:19.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:19.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.644+0000 7fbca34f9700 10 monclient: get_monmap_and_config 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.644+0000 7fbca34f9700 10 monclient: build_initial_monmap 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.644+0000 7fbca34f9700 10 monclient: monmap: 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:19.645438+0000 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:19.645438+0000 2023-09-29T23:17:19.645 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:19.646 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:19.647 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 1 Processor -- start 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 1 -- start start 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient: init 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient: _add_conns ranks=[0,9,8,3,6,2,7,1,5,10,4] 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc9c0a1f90 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:19.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc9c0a2360 0x7fbc9c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c0a2360 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 0x7fbc9c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-e con 0x7fbc9c138980 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.645+0000 7fbca34f9700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c068ed0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:19.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c09e350 con 0x7fbc9c0a2360 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c09fcf0 con 0x7fbc9c138980 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 10 monclient(hunting): authenticate will time out at 1349.006080s 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fbc9c0a2360 0x7fbc9c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1295700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fbc9c0a1f90 0x7fbc9c09f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56190/0 (socket says 172.21.15.12:56190) 2023-09-29T23:17:19.659 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1295700 1 -- 172.21.15.12:0/2722240694 learned_addr learned my addr 172.21.15.12:0/2722240694 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 0x7fbc9c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): get_auth_request con 0x7fbc9c0a2360 auth_method 0 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): get_auth_request con 0x7fbc9c138980 auth_method 0 2023-09-29T23:17:19.660 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient(hunting): handle_auth_done global_id 4914 payload 274 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 -- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 msgr2=0x7fbc9c13ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:19.661 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 0x7fbc9c13ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 -- 172.21.15.12:0/2722240694 >> v1:172.21.15.12:6789/0 conn(0x7fbc9c0a1f90 legacy=0x7fbc9c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient: _finish_hunting 0 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 monclient: found mon.noname-a 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 -- 172.21.15.12:0/2722240694 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c04c960 con 0x7fbc9c0a2360 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 1 -- 172.21.15.12:0/2722240694 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434811864 0 0) 0x7fbc9c068ed0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca1a96700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 0x7fbc9c13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca0a94700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a2360 0x7fbc9c0a27b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc980098e0 tx=0x7fbc980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73ebadb91c425b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 1 -- 172.21.15.12:0/2722240694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9800bac0 con 0x7fbc9c0a2360 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:19.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: dump: 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:19.664 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:19.665 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: _finish_auth 0 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:19.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 1 -- 172.21.15.12:0/2722240694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9800bc40 con 0x7fbc9c0a2360 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 5 monclient: authenticate success, global_id 4914 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 1 -- 172.21.15.12:0/2722240694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9800bdb0 con 0x7fbc9c0a2360 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbca34f9700 4 set_mon_vals no callback set 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:19.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.646+0000 7fbc8ffff700 10 monclient: dump: 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:19.668 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:19.669 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 10 monclient: shutdown 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 -- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a2360 msgr2=0x7fbc9c0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a2360 0x7fbc9c0a27b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc980098e0 tx=0x7fbc980091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 -- 172.21.15.12:0/2722240694 shutdown_connections 2023-09-29T23:17:19.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbc9c138980 0x7fbc9c13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 --2- 172.21.15.12:0/2722240694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a2360 0x7fbc9c0a27b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 -- 172.21.15.12:0/2722240694 >> 172.21.15.12:0/2722240694 conn(0x7fbc9c0a3b50 msgr2=0x7fbc9c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 -- 172.21.15.12:0/2722240694 shutdown_connections 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 1 -- 172.21.15.12:0/2722240694 wait complete. 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.647+0000 7fbca34f9700 10 monclient: build_initial_monmap 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.648+0000 7fbca34f9700 10 monclient: monmap: 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:19.671 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:19.672 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.648+0000 7fbca34f9700 1 Processor -- start 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.648+0000 7fbca34f9700 1 -- start start 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.648+0000 7fbca34f9700 10 monclient: init 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.648+0000 7fbca34f9700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient: _add_conns ranks=[0,5,4,6,3,7,2,1,8] 2023-09-29T23:17:19.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c0a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbc9c0a2360 0x7fbc9c097a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-h con 0x7fbc9c0a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 0x7fbc9c08c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): picked mon.noname-c con 0x7fbc9c138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:19.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c09e510 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c1430b0 con 0x7fbc9c138980 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c08fbd0 con 0x7fbc9c0a2360 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient(hunting): authenticate will time out at 1349.009121s 2023-09-29T23:17:19.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 0x7fbc9c08c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55754/0 (socket says 172.21.15.12:55754) 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 -- 172.21.15.12:0/1368931239 learned_addr learned my addr 172.21.15.12:0/1368931239 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): get_auth_request con 0x7fbc9c0a1f90 auth_method 0 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:19.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): get_auth_request con 0x7fbc9c138980 auth_method 0 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:19.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient(hunting): handle_auth_done global_id 4932 payload 824 2023-09-29T23:17:19.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 -- 172.21.15.12:0/1368931239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbc9c0a2360 msgr2=0x7fbc9c097a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbc9c0a2360 0x7fbc9c097a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 -- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 msgr2=0x7fbc9c08c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 0x7fbc9c08c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient: _finish_hunting 0 2023-09-29T23:17:19.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 monclient: found mon.noname-a 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 -- 172.21.15.12:0/1368931239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc94009a20 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1a96700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 0x7fbc9c08c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca1295700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c810 tx=0x7fbc90007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:19.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc900070f0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: dump: 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.681 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:19.682 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: _finish_auth 0 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 5 monclient: authenticate success, global_id 4932 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900072b0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient: _renew_subs 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc98009420 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc90012400 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8f7fe700 4 set_mon_vals no callback set 2023-09-29T23:17:19.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.649+0000 7fbc8d7fa700 10 monclient: dump: 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:19.686 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:19.687 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 10 monclient: _renew_subs 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c090190 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbc90005b30 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c070bc0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbc8d7fa700 1 --2- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 0x7fbc80038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:19.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.650+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbc90046e10 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.651+0000 7fbca0a94700 1 --2- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 0x7fbc80038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.651+0000 7fbca0a94700 10 monclient: get_auth_request con 0x7fbc80036850 auth_method 0 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.651+0000 7fbca0a94700 1 --2- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 0x7fbc80038d00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c08dbc0 tx=0x7fbc98017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.655+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc90012580 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.655+0000 7fbc8d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:19.690 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.655+0000 7fbc8d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:19.890 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbca34f9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbca34f9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbca34f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbc9c08ece0 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbc8d7fa700 1 -- 172.21.15.12:0/1368931239 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbc90011e90 con 0x7fbc9c0a1f90 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbc8d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.890+0000 7fbc8d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:19.891 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 msgr2=0x7fbc80038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 0x7fbc80038d00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c08dbc0 tx=0x7fbc98017000 comp rx=0 tx=0).stop 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 10 monclient: shutdown 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 msgr2=0x7fbc9c142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c810 tx=0x7fbc90007d90 comp rx=0 tx=0).stop 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 shutdown_connections 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> v2:172.21.15.153:6812/83257 conn(0x7fbc80036850 0x7fbc80038d00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbc9c0a1f90 0x7fbc9c142210 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbc9c138980 0x7fbc9c08c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 --2- 172.21.15.12:0/1368931239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fbc9c0a2360 0x7fbc9c097a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.892+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 >> 172.21.15.12:0/1368931239 conn(0x7fbc9c0a3b50 msgr2=0x7fbc9c09a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.893+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 shutdown_connections 2023-09-29T23:17:19.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:19.893+0000 7fbca34f9700 1 -- 172.21.15.12:0/1368931239 wait complete. 2023-09-29T23:17:22.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.035+0000 7fc9b1c90700 10 monclient: get_monmap_and_config 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.035+0000 7fc9b1c90700 10 monclient: build_initial_monmap 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.035+0000 7fc9b1c90700 10 monclient: monmap: 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:23.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:23.036086+0000 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:23.036086+0000 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:23.036 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:23.037 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:23.037 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:23.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 Processor -- start 2023-09-29T23:17:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 -- start start 2023-09-29T23:17:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient: init 2023-09-29T23:17:23.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient: _add_conns ranks=[7,4,0,1,9,3,6,5,10,8,2] 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc9ac0a1f90 0x7fc9ac09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-i con 0x7fc9ac0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-c con 0x7fc9ac0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9ac133740 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac068ed0 con 0x7fc9ac133740 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac09e350 con 0x7fc9ac0a2360 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:23.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac09ef70 con 0x7fc9ac0a1f90 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9b1c90700 10 monclient(hunting): authenticate will time out at 1352.396642s 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9abfff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fc9ac133740 0x7fc9ac133bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:56198/0 (socket says 172.21.15.12:56198) 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9abfff700 1 -- 172.21.15.12:0/2290144328 learned_addr learned my addr 172.21.15.12:0/2290144328 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): get_auth_request con 0x7fc9ac0a2360 auth_method 0 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:23.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9ab7fe700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc9ac0a1f90 0x7fc9ac09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9ab7fe700 10 monclient(hunting): get_auth_request con 0x7fc9ac0a1f90 auth_method 0 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.036+0000 7fc9ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_done global_id 4747 payload 274 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 -- 172.21.15.12:0/2290144328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc9ac0a1f90 msgr2=0x7fc9ac09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:23.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc9ac0a1f90 0x7fc9ac09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 -- 172.21.15.12:0/2290144328 >> v1:172.21.15.12:6789/0 conn(0x7fc9ac133740 legacy=0x7fc9ac133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 10 monclient: _finish_hunting 0 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 monclient: found mon.noname-c 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 1 -- 172.21.15.12:0/2290144328 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693757389 0 0) 0x7fc9ac068ed0 con 0x7fc9ac133740 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 -- 172.21.15.12:0/2290144328 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac04c960 con 0x7fc9ac0a2360 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aaffd700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac0a27b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009940 tx=0x7fc9a00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ddc4373682fc4d38 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 1 -- 172.21.15.12:0/2290144328 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc9a000baf0 con 0x7fc9ac0a2360 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:23.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: dump: 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:23.050 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: _finish_auth 0 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 1 -- 172.21.15.12:0/2290144328 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a000bc70 con 0x7fc9ac0a2360 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 5 monclient: authenticate success, global_id 4747 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 1 -- 172.21.15.12:0/2290144328 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc9a000bde0 con 0x7fc9ac0a2360 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 4 set_mon_vals no callback set 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:17:23.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9aa7fc700 10 monclient: dump: 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:23.053 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 10 monclient: shutdown 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 1 -- 172.21.15.12:0/2290144328 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 msgr2=0x7fc9ac0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.037+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac0a27b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009940 tx=0x7fc9a00091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 -- 172.21.15.12:0/2290144328 shutdown_connections 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fc9ac0a1f90 0x7fc9ac09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/2290144328 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac0a27b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 -- 172.21.15.12:0/2290144328 >> 172.21.15.12:0/2290144328 conn(0x7fc9ac0a3b50 msgr2=0x7fc9ac0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 -- 172.21.15.12:0/2290144328 shutdown_connections 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 1 -- 172.21.15.12:0/2290144328 wait complete. 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 10 monclient: build_initial_monmap 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.038+0000 7fc9b1c90700 10 monclient: monmap: 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:23.055 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 Processor -- start 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 -- start start 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient: init 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:23.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient: _add_conns ranks=[3,4,2,6,1,8,7,0,5] 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9ac0a1f90 0x7fc9ac142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-g con 0x7fc9ac0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-c con 0x7fc9ac0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 0x7fc9ac12d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): picked mon.noname-b con 0x7fc9ac133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac131320 con 0x7fc9ac133740 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac09ffc0 con 0x7fc9ac0a1f90 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:23.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac12fe70 con 0x7fc9ac0a2360 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9b1c90700 10 monclient(hunting): authenticate will time out at 1352.399572s 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 0x7fc9ac12d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:49642/0 (socket says 172.21.15.12:49642) 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 1 -- 172.21.15.12:0/1068347397 learned_addr learned my addr 172.21.15.12:0/1068347397 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): get_auth_request con 0x7fc9ac0a2360 auth_method 0 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): get_auth_request con 0x7fc9ac133740 auth_method 0 2023-09-29T23:17:23.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9ac0a1f90 0x7fc9ac142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): get_auth_request con 0x7fc9ac0a1f90 auth_method 0 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:23.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.039+0000 7fc9abfff700 10 monclient(hunting): handle_auth_done global_id 4619 payload 824 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 -- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 msgr2=0x7fc9ac132b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 -- 172.21.15.12:0/1068347397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9ac0a1f90 msgr2=0x7fc9ac142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9ac0a1f90 0x7fc9ac142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:23.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 monclient: found mon.noname-b 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 -- 172.21.15.12:0/1068347397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0009420 con 0x7fc9ac133740 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9aaffd700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9abfff700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 0x7fc9ac12d690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400cb90 tx=0x7fc9a4001980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc9a400ac60 con 0x7fc9ac133740 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: dump: 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:23.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:23.062 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: _finish_auth 0 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a400ade0 con 0x7fc9ac133740 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 5 monclient: authenticate success, global_id 4619 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 10 monclient: _renew_subs 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac143210 con 0x7fc9ac133740 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc9a4007570 con 0x7fc9ac133740 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:23.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9937fe700 10 monclient: dump: 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:23.064 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 10 monclient: _renew_subs 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9a9ffb700 4 set_mon_vals no callback set 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.040+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac131040 con 0x7fc9ac133740 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc9a40070f0 con 0x7fc9ac133740 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9b1c90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:23.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9b1c90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9b1c90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac070bc0 con 0x7fc9ac133740 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9937fe700 1 --2- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 0x7fc998038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc9a4047650 con 0x7fc9ac133740 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9ab7fe700 1 --2- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 0x7fc998038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.041+0000 7fc9ab7fe700 10 monclient: get_auth_request con 0x7fc998036850 auth_method 0 2023-09-29T23:17:23.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.042+0000 7fc9ab7fe700 1 --2- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 0x7fc998038d00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d10 tx=0x7fc99c0091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.046+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9a40076d0 con 0x7fc9ac133740 2023-09-29T23:17:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.046+0000 7fc9937fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:23.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.046+0000 7fc9937fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:23.284 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9b1c90700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9b1c90700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9b1c90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc9ac0a5c00 con 0x7fc9ac133740 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9937fe700 1 -- 172.21.15.12:0/1068347397 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc9a40113a0 con 0x7fc9ac133740 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9937fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.284+0000 7fc9937fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:23.285 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 msgr2=0x7fc998038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 0x7fc998038d00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d10 tx=0x7fc99c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 10 monclient: shutdown 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 msgr2=0x7fc9ac12d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 0x7fc9ac12d690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400cb90 tx=0x7fc9a4001980 comp rx=0 tx=0).stop 2023-09-29T23:17:23.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 shutdown_connections 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> v2:172.21.15.153:6812/83257 conn(0x7fc998036850 0x7fc998038d00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fc9ac133740 0x7fc9ac12d690 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9ac0a1f90 0x7fc9ac142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.286+0000 7fc9b1c90700 1 --2- 172.21.15.12:0/1068347397 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fc9ac0a2360 0x7fc9ac132b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.287+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 >> 172.21.15.12:0/1068347397 conn(0x7fc9ac0a3b50 msgr2=0x7fc9ac0a6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.287+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 shutdown_connections 2023-09-29T23:17:23.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:23.287+0000 7fc9b1c90700 1 -- 172.21.15.12:0/1068347397 wait complete. 2023-09-29T23:17:26.299 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.431+0000 7faf05448700 10 monclient: get_monmap_and_config 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.431+0000 7faf05448700 10 monclient: build_initial_monmap 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.431+0000 7faf05448700 10 monclient: monmap: 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:26.432712+0000 2023-09-29T23:17:26.432 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:26.432712+0000 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:26.433 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:26.434 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:26.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.432+0000 7faf05448700 1 Processor -- start 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.432+0000 7faf05448700 1 -- start start 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.432+0000 7faf05448700 10 monclient: init 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient: _add_conns ranks=[1,8,10,9,0,6,3,5,7,2,4] 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7faf00133130 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): picked mon.noname-e con 0x7faf00133c10 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7faf00138670 0x7faf0013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): picked mon.noname-f con 0x7faf00138670 addr v2:172.21.15.153:3300/0 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf00068ed0 con 0x7faf00133130 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0009b2a0 con 0x7faf00138670 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7faf000a27a0 con 0x7faf00133c10 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf05448700 10 monclient(hunting): authenticate will time out at 1355.793283s 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40826/0 (socket says 172.21.15.12:40826) 2023-09-29T23:17:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 -- 172.21.15.12:0/3582181103 learned_addr learned my addr 172.21.15.12:0/3582181103 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): get_auth_request con 0x7faf00133c10 auth_method 0 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 1 --2- 172.21.15.12:0/3582181103 >> v2:172.21.15.153:3300/0 conn(0x7faf00138670 0x7faf0013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): get_auth_request con 0x7faf00138670 auth_method 0 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:26.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient(hunting): handle_auth_done global_id 4634 payload 274 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 -- 172.21.15.12:0/3582181103 >> v2:172.21.15.153:3300/0 conn(0x7faf00138670 msgr2=0x7faf0013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 --2- 172.21.15.12:0/3582181103 >> v2:172.21.15.153:3300/0 conn(0x7faf00138670 0x7faf0013aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 -- 172.21.15.12:0/3582181103 >> v1:172.21.15.153:6789/0 conn(0x7faf00133130 legacy=0x7faf00133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 monclient: found mon.noname-e 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faefeffd700 1 -- 172.21.15.12:0/3582181103 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255926085 0 0) 0x7faf00068ed0 con 0x7faf00133130 2023-09-29T23:17:26.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 -- 172.21.15.12:0/3582181103 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0004c960 con 0x7faf00133c10 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faefeffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.433+0000 7faeff7fe700 1 --2- 172.21.15.12:0/3582181103 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faef00099e0 tx=0x7faef0009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b8f42f78eae36ee2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 1 -- 172.21.15.12:0/3582181103 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7faef000bca0 con 0x7faf00133c10 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: dump: 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:26.446 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:26.447 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: _finish_auth 0 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 1 -- 172.21.15.12:0/3582181103 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef000be20 con 0x7faf00133c10 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 5 monclient: authenticate success, global_id 4634 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 1 -- 172.21.15.12:0/3582181103 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7faef00118a0 con 0x7faf00133c10 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 4 set_mon_vals no callback set 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:26.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faefeffd700 10 monclient: dump: 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:26.449 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 10 monclient: shutdown 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 -- 172.21.15.12:0/3582181103 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 msgr2=0x7faf000a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 --2- 172.21.15.12:0/3582181103 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faef00099e0 tx=0x7faef0009250 comp rx=0 tx=0).stop 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 -- 172.21.15.12:0/3582181103 shutdown_connections 2023-09-29T23:17:26.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 --2- 172.21.15.12:0/3582181103 >> v2:172.21.15.153:3300/0 conn(0x7faf00138670 0x7faf0013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 --2- 172.21.15.12:0/3582181103 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf000a0e80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 -- 172.21.15.12:0/3582181103 >> 172.21.15.12:0/3582181103 conn(0x7faf000a6490 msgr2=0x7faf000a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 -- 172.21.15.12:0/3582181103 shutdown_connections 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.434+0000 7faf05448700 1 -- 172.21.15.12:0/3582181103 wait complete. 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.435+0000 7faf05448700 10 monclient: build_initial_monmap 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.435+0000 7faf05448700 10 monclient: monmap: 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:26.451 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.435+0000 7faf05448700 1 Processor -- start 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.435+0000 7faf05448700 1 -- start start 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.435+0000 7faf05448700 10 monclient: init 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient: _add_conns ranks=[3,8,4,7,0,6,2,5,1] 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf00133130 0x7faf00141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): picked mon.noname-g con 0x7faf00133130 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): picked mon.noname-e con 0x7faf00133c10 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7faf00138670 0x7faf0009ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): picked mon.noname-c con 0x7faf00138670 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:26.453 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf001359e0 con 0x7faf00133130 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7faf0009b020 con 0x7faf00138670 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7faf00134430 con 0x7faf00133c10 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf05448700 10 monclient(hunting): authenticate will time out at 1355.796302s 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40832/0 (socket says 172.21.15.12:40832) 2023-09-29T23:17:26.454 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 -- 172.21.15.12:0/341867960 learned_addr learned my addr 172.21.15.12:0/341867960 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): get_auth_request con 0x7faf00133c10 auth_method 0 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7faf00138670 0x7faf0009ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): get_auth_request con 0x7faf00138670 auth_method 0 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:26.455 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf00133130 0x7faf00141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): get_auth_request con 0x7faf00133130 auth_method 0 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faf04c47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faefffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:26.456 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient(hunting): handle_auth_done global_id 4643 payload 824 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 -- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7faf00138670 msgr2=0x7faf0009ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7faf00138670 0x7faf0009ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 -- 172.21.15.12:0/341867960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf00133130 msgr2=0x7faf00141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf00133130 0x7faf00141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 monclient: found mon.noname-e 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 -- 172.21.15.12:0/341867960 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef00094f0 con 0x7faf00133c10 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.436+0000 7faeff7fe700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf000a28e0 tx=0x7faef0017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7faef00118a0 con 0x7faf00133c10 2023-09-29T23:17:26.457 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: dump: 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:26.458 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: _finish_auth 0 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 5 monclient: authenticate success, global_id 4643 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef0011a60 con 0x7faf00133c10 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:26.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7faef0011bd0 con 0x7faf00133c10 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 10 monclient: _renew_subs 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0009b3e0 con 0x7faf00133c10 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faefe7fc700 4 set_mon_vals no callback set 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 10 monclient: dump: 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:26.460 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:26.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 10 monclient: _renew_subs 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf000a06e0 con 0x7faf00133c10 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.437+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7faef0011d50 con 0x7faf00133c10 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faf05448700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faf05448700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faf05448700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf00067740 con 0x7faf00133c10 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faee7fff700 1 --2- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 0x7faeec038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7faef00486b0 con 0x7faf00133c10 2023-09-29T23:17:26.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faefffff700 1 --2- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 0x7faeec038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faefffff700 10 monclient: get_auth_request con 0x7faeec036850 auth_method 0 2023-09-29T23:17:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.438+0000 7faefffff700 1 --2- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 0x7faeec038d00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faef40039f0 tx=0x7faef4007ea0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.443+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faef0011420 con 0x7faf00133c10 2023-09-29T23:17:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.443+0000 7faee7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.443+0000 7faee7fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.676+0000 7faf05448700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:26.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.676+0000 7faf05448700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.676+0000 7faf05448700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.676+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faf00070bc0 con 0x7faf00133c10 2023-09-29T23:17:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.677+0000 7faee7fff700 1 -- 172.21.15.12:0/341867960 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7faef0028080 con 0x7faf00133c10 2023-09-29T23:17:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.677+0000 7faee7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:26.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.677+0000 7faee7fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:26.678 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:26.678 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 msgr2=0x7faeec038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 0x7faeec038d00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faef40039f0 tx=0x7faef4007ea0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 10 monclient: shutdown 2023-09-29T23:17:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:26.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 msgr2=0x7faf001377b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf000a28e0 tx=0x7faef0017000 comp rx=0 tx=0).stop 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 shutdown_connections 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7faf00138670 0x7faf0009ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> v2:172.21.15.153:6812/83257 conn(0x7faeec036850 0x7faeec038d00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf00133130 0x7faf00141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 --2- 172.21.15.12:0/341867960 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7faf00133c10 0x7faf001377b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 >> 172.21.15.12:0/341867960 conn(0x7faf000a6490 msgr2=0x7faf000a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 shutdown_connections 2023-09-29T23:17:26.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:26.679+0000 7faf05448700 1 -- 172.21.15.12:0/341867960 wait complete. 2023-09-29T23:17:29.691 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 10 monclient: get_monmap_and_config 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 10 monclient: build_initial_monmap 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 10 monclient: monmap: 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:29.821 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:29.822018+0000 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:29.822018+0000 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:29.822 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:29.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 1 Processor -- start 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 1 -- start start 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.821+0000 7fd722277700 10 monclient: init 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient: _add_conns ranks=[8,4,1,5,6,0,3,10,7,2,9] 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): picked mon.noname-e con 0x7fd71c0a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:29.832 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): picked mon.noname-c con 0x7fd71c0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd71c133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c068ed0 con 0x7fd71c133740 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c09e350 con 0x7fd71c0a2360 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c09ef70 con 0x7fd71c0a1f90 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd722277700 10 monclient(hunting): authenticate will time out at 1359.182461s 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.833 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:46036/0 (socket says 172.21.15.12:46036) 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40842/0 (socket says 172.21.15.12:40842) 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 -- 172.21.15.12:0/2961910064 learned_addr learned my addr 172.21.15.12:0/2961910064 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): get_auth_request con 0x7fd71c0a2360 auth_method 0 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): get_auth_request con 0x7fd71c0a1f90 auth_method 0 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:29.834 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_done global_id 4774 payload 274 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 -- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 msgr2=0x7fd71c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 -- 172.21.15.12:0/2961910064 >> v1:172.21.15.153:6789/0 conn(0x7fd71c133740 legacy=0x7fd71c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 monclient: found mon.noname-c 2023-09-29T23:17:29.835 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71b7fe700 1 -- 172.21.15.12:0/2961910064 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c04c960 con 0x7fd71c0a2360 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.822+0000 7fd71bfff700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71b7fe700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd710009a10 tx=0x7fd710009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8eb5d158cab16ca8 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 1 -- 172.21.15.12:0/2961910064 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd71000bc60 con 0x7fd71c0a2360 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: dump: 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.836 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:29.837 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: _finish_auth 0 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 1 -- 172.21.15.12:0/2961910064 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71000bde0 con 0x7fd71c0a2360 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 5 monclient: authenticate success, global_id 4774 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 1 -- 172.21.15.12:0/2961910064 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd7100118a0 con 0x7fd71c0a2360 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:29.838 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 4 set_mon_vals no callback set 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd71affd700 10 monclient: dump: 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:29.839 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 10 monclient: shutdown 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 -- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 msgr2=0x7fd71c0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd710009a10 tx=0x7fd710009250 comp rx=0 tx=0).stop 2023-09-29T23:17:29.840 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 -- 172.21.15.12:0/2961910064 shutdown_connections 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fd71c0a1f90 0x7fd71c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 --2- 172.21.15.12:0/2961910064 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fd71c0a2360 0x7fd71c0a27b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 -- 172.21.15.12:0/2961910064 >> 172.21.15.12:0/2961910064 conn(0x7fd71c0a3b50 msgr2=0x7fd71c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 -- 172.21.15.12:0/2961910064 shutdown_connections 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.823+0000 7fd722277700 1 -- 172.21.15.12:0/2961910064 wait complete. 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.824+0000 7fd722277700 10 monclient: build_initial_monmap 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.824+0000 7fd722277700 10 monclient: monmap: 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:29.841 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.842 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.824+0000 7fd722277700 1 Processor -- start 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.824+0000 7fd722277700 1 -- start start 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.824+0000 7fd722277700 10 monclient: init 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient: _add_conns ranks=[6,0,3,7,8,2,4,5,1] 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): picked mon.noname-d con 0x7fd71c0a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd71c0a2360 0x7fd71c132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): picked mon.noname-a con 0x7fd71c0a2360 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd71c133740 0x7fd71c12d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): picked mon.noname-g con 0x7fd71c133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:29.843 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c09e510 con 0x7fd71c0a2360 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c1311a0 con 0x7fd71c133740 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c130b50 con 0x7fd71c0a1f90 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd722277700 10 monclient(hunting): authenticate will time out at 1359.185463s 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd71c0a2360 0x7fd71c132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35286/0 (socket says 172.21.15.12:35286) 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 -- 172.21.15.12:0/3576372605 learned_addr learned my addr 172.21.15.12:0/3576372605 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:29.844 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): get_auth_request con 0x7fd71c0a1f90 auth_method 0 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): get_auth_request con 0x7fd71c0a2360 auth_method 0 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:29.845 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd71c133740 0x7fd71c12d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): get_auth_request con 0x7fd71c133740 auth_method 0 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd720814700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient(hunting): handle_auth_done global_id 4758 payload 824 2023-09-29T23:17:29.846 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 -- 172.21.15.12:0/3576372605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd71c133740 msgr2=0x7fd71c12d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd71c133740 0x7fd71c12d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 -- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd71c0a2360 msgr2=0x7fd71c132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd71c0a2360 0x7fd71c132920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 monclient: found mon.noname-d 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.825+0000 7fd71bfff700 1 -- 172.21.15.12:0/3576372605 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c009420 con 0x7fd71c0a1f90 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd71bfff700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00ec80 tx=0x7fd70c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd70c00ccd0 con 0x7fd71c0a1f90 2023-09-29T23:17:29.847 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: dump: 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:29.848 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: _finish_auth 0 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 5 monclient: authenticate success, global_id 4758 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 10 monclient: _renew_subs 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:29.849 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7100094f0 con 0x7fd71c0a1f90 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 10 monclient: _renew_subs 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c12fc60 con 0x7fd71c0a1f90 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c00ce30 con 0x7fd71c0a1f90 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd70c005a00 con 0x7fd71c0a1f90 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd71a7fc700 4 set_mon_vals no callback set 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.826+0000 7fd703fff700 10 monclient: dump: 2023-09-29T23:17:29.850 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:29.851 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd70c00c850 con 0x7fd71c0a1f90 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd722277700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd722277700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd722277700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc0050b0 con 0x7fd71c0a1f90 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd703fff700 1 --2- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 0x7fd708038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd70c0479b0 con 0x7fd71c0a1f90 2023-09-29T23:17:29.852 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd71b7fe700 1 --2- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 0x7fd708038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.827+0000 7fd71b7fe700 10 monclient: get_auth_request con 0x7fd708036850 auth_method 0 2023-09-29T23:17:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.828+0000 7fd71b7fe700 1 --2- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 0x7fd708038d00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd710009fd0 tx=0x7fd710016000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.833+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd70c01adc0 con 0x7fd71c0a1f90 2023-09-29T23:17:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.833+0000 7fd703fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:29.853 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:29.833+0000 7fd703fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd722277700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd722277700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:30.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd722277700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd6fc005360 con 0x7fd71c0a1f90 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd703fff700 1 -- 172.21.15.12:0/3576372605 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd70c01a360 con 0x7fd71c0a1f90 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd703fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:30.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.068+0000 7fd703fff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 msgr2=0x7fd708038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 0x7fd708038d00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd710009fd0 tx=0x7fd710016000 comp rx=0 tx=0).stop 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 10 monclient: shutdown 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 msgr2=0x7fd71c142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00ec80 tx=0x7fd70c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 shutdown_connections 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> v2:172.21.15.153:6812/83257 conn(0x7fd708036850 0x7fd708038d00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd71c133740 0x7fd71c12d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:30.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fd71c0a1f90 0x7fd71c142210 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 --2- 172.21.15.12:0/3576372605 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd71c0a2360 0x7fd71c132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 >> 172.21.15.12:0/3576372605 conn(0x7fd71c0a3b50 msgr2=0x7fd71c135d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 shutdown_connections 2023-09-29T23:17:30.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:30.070+0000 7fd722277700 1 -- 172.21.15.12:0/3576372605 wait complete. 2023-09-29T23:17:33.083 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.215+0000 7feb0f59e700 10 monclient: get_monmap_and_config 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.215+0000 7feb0f59e700 10 monclient: build_initial_monmap 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.215+0000 7feb0f59e700 10 monclient: monmap: 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:33.216583+0000 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:33.216583+0000 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:33.217 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.216+0000 7feb0f59e700 1 Processor -- start 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.216+0000 7feb0f59e700 1 -- start start 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.216+0000 7feb0f59e700 10 monclient: init 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.216+0000 7feb0f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.216+0000 7feb0f59e700 10 monclient: _add_conns ranks=[9,8,1,4,0,7,5,10,6,2,3] 2023-09-29T23:17:33.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb10133130 0x7feb100a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-a con 0x7feb10133130 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-e con 0x7feb10133500 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7feb1012fc20 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10068ed0 con 0x7feb1012fc20 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb100a27a0 con 0x7feb10133130 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7feb1009adf0 con 0x7feb10133500 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:33.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 10 monclient(hunting): authenticate will time out at 1362.577098s 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb10133130 0x7feb100a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40856/0 (socket says 172.21.15.12:40856) 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 1 -- 172.21.15.12:0/1161352542 learned_addr learned my addr 172.21.15.12:0/1161352542 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): get_auth_request con 0x7feb10133500 auth_method 0 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7feb10133130 0x7feb100a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35270/0 (socket says 172.21.15.12:35270) 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): get_auth_request con 0x7feb10133130 auth_method 0 2023-09-29T23:17:33.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient(hunting): handle_auth_done global_id 4959 payload 274 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 -- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 msgr2=0x7feb10133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:33.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 -- 172.21.15.12:0/1161352542 >> v1:172.21.15.153:6789/0 conn(0x7feb1012fc20 legacy=0x7feb101300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient: _finish_hunting 0 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 monclient: found mon.noname-a 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 -- 172.21.15.12:0/1161352542 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1004c960 con 0x7feb10133130 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0dd9b700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0e59c700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10133130 0x7feb100a0e80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb000098e0 tx=0x7feb000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbe8de06fb1b644 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 1 -- 172.21.15.12:0/1161352542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb0000bac0 con 0x7feb10133130 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:33.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 10 monclient: dump: 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:33.231 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 10 monclient: _finish_auth 0 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 5 monclient: authenticate success, global_id 4959 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 1 -- 172.21.15.12:0/1161352542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0000bc40 con 0x7feb10133130 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:33.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.217+0000 7feb0d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0d59a700 1 -- 172.21.15.12:0/1161352542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feb0000bdb0 con 0x7feb10133130 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 4 set_mon_vals no callback set 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0d59a700 10 monclient: dump: 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:33.233 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 10 monclient: shutdown 2023-09-29T23:17:33.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 -- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10133130 msgr2=0x7feb100a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10133130 0x7feb100a0e80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb000098e0 tx=0x7feb000091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 -- 172.21.15.12:0/1161352542 shutdown_connections 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb10133130 0x7feb100a0e80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 --2- 172.21.15.12:0/1161352542 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133500 0x7feb10133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 -- 172.21.15.12:0/1161352542 >> 172.21.15.12:0/1161352542 conn(0x7feb100a6490 msgr2=0x7feb100a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 -- 172.21.15.12:0/1161352542 shutdown_connections 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.218+0000 7feb0f59e700 1 -- 172.21.15.12:0/1161352542 wait complete. 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient: build_initial_monmap 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient: monmap: 2023-09-29T23:17:33.235 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:33.236 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 Processor -- start 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 -- start start 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient: init 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient: _add_conns ranks=[5,8,7,3,4,2,1,6,0] 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feb1012fc20 0x7feb1012eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-h con 0x7feb1012fc20 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:33.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-e con 0x7feb10133130 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb10133500 0x7feb1006dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): picked mon.noname-i con 0x7feb10133500 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7feb1006f700 con 0x7feb1012fc20 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feb1011dcd0 con 0x7feb10133500 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7feb1011cde0 con 0x7feb10133130 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:33.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.219+0000 7feb0f59e700 10 monclient(hunting): authenticate will time out at 1362.579943s 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40866/0 (socket says 172.21.15.12:40866) 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 -- 172.21.15.12:0/266025212 learned_addr learned my addr 172.21.15.12:0/266025212 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): get_auth_request con 0x7feb10133130 auth_method 0 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feb1012fc20 0x7feb1012eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:33.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb10133500 0x7feb1006dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): get_auth_request con 0x7feb1012fc20 auth_method 0 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): get_auth_request con 0x7feb10133500 auth_method 0 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:33.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient(hunting): handle_auth_done global_id 4670 payload 824 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 -- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb10133500 msgr2=0x7feb1006dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb10133500 0x7feb1006dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 -- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feb1012fc20 msgr2=0x7feb1012eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feb1012fc20 0x7feb1012eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient: _finish_hunting 0 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 monclient: found mon.noname-e 2023-09-29T23:17:33.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 -- 172.21.15.12:0/266025212 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80094f0 con 0x7feb10133130 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0dd9b700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feaf800ec50 tx=0x7feaf8009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feaf80097d0 con 0x7feb10133130 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: dump: 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.242 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:33.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: _finish_auth 0 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 5 monclient: authenticate success, global_id 4670 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 10 monclient: _renew_subs 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf800ccd0 con 0x7feb10133130 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb00009420 con 0x7feb10133130 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feaf800ce40 con 0x7feb10133130 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:33.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0cd99700 4 set_mon_vals no callback set 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb067fc700 10 monclient: dump: 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:33.245 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 10 monclient: _renew_subs 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.220+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1009b020 con 0x7feb10133130 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7feaf800c850 con 0x7feb10133130 2023-09-29T23:17:33.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0f59e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb067fc700 1 --2- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 0x7feaf0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1011e1d0 con 0x7feb10133130 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7feaf8047510 con 0x7feb10133130 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0e59c700 1 --2- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 0x7feaf0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.221+0000 7feb0e59c700 10 monclient: get_auth_request con 0x7feaf0036850 auth_method 0 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.222+0000 7feb0e59c700 1 --2- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 0x7feaf0038d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb000098e0 tx=0x7feb00017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.226+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb1011e1d0 con 0x7feb10133130 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.226+0000 7feb067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:33.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.226+0000 7feb067fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb0f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb0f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb0f59e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feb1011e1d0 con 0x7feb10133130 2023-09-29T23:17:33.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb067fc700 1 -- 172.21.15.12:0/266025212 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7feb1011e1d0 con 0x7feb10133130 2023-09-29T23:17:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:33.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.464+0000 7feb067fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:33.465 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:33.465 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:33.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 msgr2=0x7feaf0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 0x7feaf0038d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb000098e0 tx=0x7feb00017000 comp rx=0 tx=0).stop 2023-09-29T23:17:33.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 10 monclient: shutdown 2023-09-29T23:17:33.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:33.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 msgr2=0x7feb1006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:33.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feaf800ec50 tx=0x7feaf8009250 comp rx=0 tx=0).stop 2023-09-29T23:17:33.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 shutdown_connections 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> v2:172.21.15.153:6812/83257 conn(0x7feaf0036850 0x7feaf0038d00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7feb10133500 0x7feb1006dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7feb1012fc20 0x7feb1012eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 --2- 172.21.15.12:0/266025212 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7feb10133130 0x7feb1006d570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 >> 172.21.15.12:0/266025212 conn(0x7feb100a6490 msgr2=0x7feb100a5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:33.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 shutdown_connections 2023-09-29T23:17:33.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:33.466+0000 7feb0f59e700 1 -- 172.21.15.12:0/266025212 wait complete. 2023-09-29T23:17:36.481 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 10 monclient: get_monmap_and_config 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 10 monclient: build_initial_monmap 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 10 monclient: monmap: 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:36.613 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:36.614020+0000 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:36.614020+0000 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:36.614 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:36.615 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:36.615 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:36.615 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:36.615 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.622 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 1 Processor -- start 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 1 -- start start 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.613+0000 7f0e7ea3e700 10 monclient: init 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient: _add_conns ranks=[5,3,9,0,7,6,1,10,8,2,4] 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-h con 0x7f0e7806eeb0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e7806f280 0x7f0e780698a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-g con 0x7f0e7806f280 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0e7806f6b0 0x7f0e78069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-a con 0x7f0e7806f6b0 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7809fad0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78133640 con 0x7f0e7806f280 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0e780679e0 con 0x7f0e7806eeb0 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7ea3e700 10 monclient(hunting): authenticate will time out at 1365.974453s 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0e7806f6b0 0x7f0e78069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f0e7806f6b0 0x7f0e78069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41914/0 (socket says 172.21.15.12:41914) 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/905651954 learned_addr learned my addr 172.21.15.12:0/905651954 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): get_auth_request con 0x7f0e7806f6b0 auth_method 0 2023-09-29T23:17:36.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e777fe700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e7806f280 0x7f0e780698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e77fff700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e777fe700 10 monclient(hunting): get_auth_request con 0x7f0e7806f280 auth_method 0 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e777fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e77fff700 10 monclient(hunting): get_auth_request con 0x7f0e7806eeb0 auth_method 0 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e77fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_done global_id 4968 payload 274 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 msgr2=0x7f0e78133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78133aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e7806f280 msgr2=0x7f0e780698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e7806f280 0x7f0e780698a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient: _finish_hunting 0 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 monclient: found mon.noname-a 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:36.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.614+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/905651954 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78068ed0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e7806f6b0 0x7f0e78069de0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009f20 tx=0x7f0e70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28029a0ce596e1c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 1 -- 172.21.15.12:0/905651954 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e7000e070 con 0x7f0e7806f6b0 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: dump: 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:36.627 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: _finish_auth 0 2023-09-29T23:17:36.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 1 -- 172.21.15.12:0/905651954 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70007400 con 0x7f0e7806f6b0 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 5 monclient: authenticate success, global_id 4968 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 4 set_mon_vals no callback set 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 1 -- 172.21.15.12:0/905651954 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e700092e0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e76ffd700 10 monclient: dump: 2023-09-29T23:17:36.629 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:36.630 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 10 monclient: shutdown 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/905651954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e7806f6b0 msgr2=0x7f0e78069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e7806f6b0 0x7f0e78069de0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009f20 tx=0x7f0e70009840 comp rx=0 tx=0).stop 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/905651954 shutdown_connections 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e7806f6b0 0x7f0e78069de0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/905651954 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e7806f280 0x7f0e780698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/905651954 >> 172.21.15.12:0/905651954 conn(0x7f0e7806aab0 msgr2=0x7f0e7806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/905651954 shutdown_connections 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.615+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/905651954 wait complete. 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.616+0000 7f0e7ea3e700 10 monclient: build_initial_monmap 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.616+0000 7f0e7ea3e700 10 monclient: monmap: 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:36.632 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.616+0000 7f0e7ea3e700 1 Processor -- start 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 -- start start 2023-09-29T23:17:36.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient: init 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient: _add_conns ranks=[5,4,2,8,7,0,6,1,3] 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-h con 0x7f0e7806eeb0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-c con 0x7f0e7806f280 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 0x7f0e7811dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): picked mon.noname-b con 0x7f0e7806f6b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78134870 con 0x7f0e7806f6b0 2023-09-29T23:17:36.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7811f550 con 0x7f0e7806f280 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781208c0 con 0x7f0e7806eeb0 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7ea3e700 10 monclient(hunting): authenticate will time out at 1365.977659s 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 0x7f0e7811dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44108/0 (socket says 172.21.15.12:44108) 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 1 -- 172.21.15.12:0/3743334773 learned_addr learned my addr 172.21.15.12:0/3743334773 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): get_auth_request con 0x7f0e7806f6b0 auth_method 0 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): get_auth_request con 0x7f0e7806f280 auth_method 0 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.617+0000 7f0e77fff700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e77fff700 10 monclient(hunting): get_auth_request con 0x7f0e7806eeb0 auth_method 0 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e77fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 10 monclient(hunting): handle_auth_done global_id 4628 payload 824 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/3743334773 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 msgr2=0x7f0e78142e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78142e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 msgr2=0x7f0e78130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 10 monclient: _finish_hunting 0 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 monclient: found mon.noname-b 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:36.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 -- 172.21.15.12:0/3743334773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70009ae0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e777fe700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7cfdb700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 0x7f0e7811dda0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e70015000 tx=0x7f0e70018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e70007c80 con 0x7f0e7806f6b0 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: dump: 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.638 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:36.639 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: _finish_auth 0 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70007e40 con 0x7f0e7806f6b0 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 5 monclient: authenticate success, global_id 4628 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 10 monclient: _renew_subs 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7804c960 con 0x7f0e7806f6b0 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e700268e0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e5ffff700 10 monclient: dump: 2023-09-29T23:17:36.640 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:36.641 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e767fc700 4 set_mon_vals no callback set 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 10 monclient: _renew_subs 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.618+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e78099b10 con 0x7f0e7806f6b0 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0e70026460 con 0x7f0e7806f6b0 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e7ea3e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e7ea3e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:36.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e7ea3e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e78120fa0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e5ffff700 1 --2- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 0x7f0e64038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0e70048df0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e77fff700 1 --2- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 0x7f0e64038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.619+0000 7f0e77fff700 10 monclient: get_auth_request con 0x7f0e64036880 auth_method 0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.620+0000 7f0e77fff700 1 --2- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 0x7f0e64038d30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c006fd0 tx=0x7f0e6c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:36.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.624+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e7001c9a0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.624+0000 7f0e5ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:36.644 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.624+0000 7f0e5ffff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e7ea3e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e7ea3e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:36.864 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e7ea3e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e78132cd0 con 0x7f0e7806f6b0 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e5ffff700 1 -- 172.21.15.12:0/3743334773 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0e70021070 con 0x7f0e7806f6b0 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e5ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.864+0000 7f0e5ffff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:36.865 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 msgr2=0x7f0e64038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 0x7f0e64038d30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c006fd0 tx=0x7f0e6c008040 comp rx=0 tx=0).stop 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 10 monclient: shutdown 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 msgr2=0x7f0e7811dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.866+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 0x7f0e7811dda0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e70015000 tx=0x7f0e70018000 comp rx=0 tx=0).stop 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 shutdown_connections 2023-09-29T23:17:36.867 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> v2:172.21.15.153:6812/83257 conn(0x7f0e64036880 0x7f0e64038d30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0e7806f6b0 0x7f0e7811dda0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0e7806eeb0 0x7f0e78142e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 --2- 172.21.15.12:0/3743334773 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0e7806f280 0x7f0e78130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 >> 172.21.15.12:0/3743334773 conn(0x7f0e7806aab0 msgr2=0x7f0e78131df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 shutdown_connections 2023-09-29T23:17:36.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:36.867+0000 7f0e7ea3e700 1 -- 172.21.15.12:0/3743334773 wait complete. 2023-09-29T23:17:39.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:40.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.006+0000 7f162a3f7700 10 monclient: get_monmap_and_config 2023-09-29T23:17:40.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.006+0000 7f162a3f7700 10 monclient: build_initial_monmap 2023-09-29T23:17:40.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.006+0000 7f162a3f7700 10 monclient: monmap: 2023-09-29T23:17:40.007 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:40.007 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:40.007615+0000 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:40.007615+0000 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:40.008 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.007+0000 7f162a3f7700 1 Processor -- start 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.007+0000 7f162a3f7700 1 -- start start 2023-09-29T23:17:40.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.007+0000 7f162a3f7700 10 monclient: init 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.007+0000 7f162a3f7700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.007+0000 7f162a3f7700 10 monclient: _add_conns ranks=[4,10,1,3,9,2,8,7,5,0,6] 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-c con 0x7f162409fad0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f162409cc70 0x7f162409d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-f con 0x7f162409cc70 addr v2:172.21.15.153:3300/0 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f16240a0e80 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1624068ed0 con 0x7f16240a0e80 2023-09-29T23:17:40.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162409ea90 con 0x7f162409cc70 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f162409f2d0 con 0x7f162409fad0 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient(hunting): authenticate will time out at 1369.368077s 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:44120/0 (socket says 172.21.15.12:44120) 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 -- 172.21.15.12:0/2738775753 learned_addr learned my addr 172.21.15.12:0/2738775753 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): get_auth_request con 0x7f162409fad0 auth_method 0 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:40.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1628bf4700 1 --2- 172.21.15.12:0/2738775753 >> v2:172.21.15.153:3300/0 conn(0x7f162409cc70 0x7f162409d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1628bf4700 10 monclient(hunting): get_auth_request con 0x7f162409cc70 auth_method 0 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1628bf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1628bf4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1628bf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient(hunting): handle_auth_done global_id 4792 payload 274 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 -- 172.21.15.12:0/2738775753 >> v2:172.21.15.153:3300/0 conn(0x7f162409cc70 msgr2=0x7f162409d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:40.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 --2- 172.21.15.12:0/2738775753 >> v2:172.21.15.153:3300/0 conn(0x7f162409cc70 0x7f162409d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 -- 172.21.15.12:0/2738775753 >> v1:172.21.15.153:6789/0 conn(0x7f16240a0e80 legacy=0x7f16240a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient: _finish_hunting 0 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 monclient: found mon.noname-c 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 -- 172.21.15.12:0/2738775753 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162404c960 con 0x7f162409fad0 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f16293f5700 1 --2- 172.21.15.12:0/2738775753 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1614009f20 tx=0x7f1614009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=24f7c1ad77bb32e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 1 -- 172.21.15.12:0/2738775753 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f161400e070 con 0x7f162409fad0 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:40.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: dump: 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:40.021 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: _finish_auth 0 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 5 monclient: authenticate success, global_id 4792 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 1 -- 172.21.15.12:0/2738775753 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1614007400 con 0x7f162409fad0 2023-09-29T23:17:40.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 1 -- 172.21.15.12:0/2738775753 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f16140092e0 con 0x7f162409fad0 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f162a3f7700 4 set_mon_vals no callback set 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.008+0000 7f1623fff700 10 monclient: dump: 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.023 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:40.024 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 10 monclient: shutdown 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 -- 172.21.15.12:0/2738775753 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 msgr2=0x7f162409fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2738775753 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1614009f20 tx=0x7f1614009840 comp rx=0 tx=0).stop 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 -- 172.21.15.12:0/2738775753 shutdown_connections 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2738775753 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f162409fad0 0x7f162409fea0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2738775753 >> v2:172.21.15.153:3300/0 conn(0x7f162409cc70 0x7f162409d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 -- 172.21.15.12:0/2738775753 >> 172.21.15.12:0/2738775753 conn(0x7f1624063cc0 msgr2=0x7f16240a6490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 -- 172.21.15.12:0/2738775753 shutdown_connections 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 1 -- 172.21.15.12:0/2738775753 wait complete. 2023-09-29T23:17:40.025 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 10 monclient: build_initial_monmap 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.009+0000 7f162a3f7700 10 monclient: monmap: 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:40.026 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 Processor -- start 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 -- start start 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient: init 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:40.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient: _add_conns ranks=[8,0,1,7,3,4,2,6,5] 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-e con 0x7f162409cc70 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f162409fad0 0x7f16241422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-a con 0x7f162409fad0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16240a0e80 0x7f1624142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): picked mon.noname-f con 0x7f16240a0e80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162406f210 con 0x7f162409fad0 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:40.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624127950 con 0x7f16240a0e80 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1624127670 con 0x7f162409cc70 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f162a3f7700 10 monclient(hunting): authenticate will time out at 1369.370908s 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.010+0000 7f16293f5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:40426/0 (socket says 172.21.15.12:40426) 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 -- 172.21.15.12:0/2000230504 learned_addr learned my addr 172.21.15.12:0/2000230504 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): get_auth_request con 0x7f162409cc70 auth_method 0 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:40.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f162409fad0 0x7f16241422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): get_auth_request con 0x7f162409fad0 auth_method 0 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16240a0e80 0x7f1624142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): get_auth_request con 0x7f16240a0e80 auth_method 0 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:40.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1628bf4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f1629bf6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient(hunting): handle_auth_done global_id 4679 payload 824 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 -- 172.21.15.12:0/2000230504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16240a0e80 msgr2=0x7f1624142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16240a0e80 0x7f1624142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.031 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 -- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f162409fad0 msgr2=0x7f16241422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f162409fad0 0x7f16241422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient: _finish_hunting 0 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 monclient: found mon.noname-e 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 -- 172.21.15.12:0/2000230504 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1614009ae0 con 0x7f162409cc70 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16293f5700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1614003000 tx=0x7f1614017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1614007190 con 0x7f162409cc70 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: dump: 2023-09-29T23:17:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: _finish_auth 0 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 5 monclient: authenticate success, global_id 4679 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1614007c00 con 0x7f162409cc70 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1614007db0 con 0x7f162409cc70 2023-09-29T23:17:40.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16217fa700 10 monclient: dump: 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f16237fe700 4 set_mon_vals no callback set 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 10 monclient: _renew_subs 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1624127390 con 0x7f162409cc70 2023-09-29T23:17:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 10 monclient: _renew_subs 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.011+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16241470d0 con 0x7f162409cc70 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1614007780 con 0x7f162409cc70 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f162a3f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f16217fa700 1 --2- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 0x7f1610038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f162a3f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f162a3f7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162409bf40 con 0x7f162409cc70 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.012+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f16140484f0 con 0x7f162409cc70 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.013+0000 7f1628bf4700 1 --2- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 0x7f1610038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.013+0000 7f1628bf4700 10 monclient: get_auth_request con 0x7f16100367b0 auth_method 0 2023-09-29T23:17:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.013+0000 7f1628bf4700 1 --2- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 0x7f1610038c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1618006fd0 tx=0x7f1618008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.017+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1614027460 con 0x7f162409cc70 2023-09-29T23:17:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.017+0000 7f16217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:40.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.017+0000 7f16217fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:40.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.251+0000 7f162a3f7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:40.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.251+0000 7f162a3f7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:40.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.251+0000 7f162a3f7700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:40.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.251+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16241470d0 con 0x7f162409cc70 2023-09-29T23:17:40.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.252+0000 7f16217fa700 1 -- 172.21.15.12:0/2000230504 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f16241470d0 con 0x7f162409cc70 2023-09-29T23:17:40.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.252+0000 7f16217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:40.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.252+0000 7f16217fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:40.254 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:40.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 msgr2=0x7f1610038c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:40.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 0x7f1610038c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1618006fd0 tx=0x7f1618008040 comp rx=0 tx=0).stop 2023-09-29T23:17:40.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 10 monclient: shutdown 2023-09-29T23:17:40.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:40.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 msgr2=0x7f162409b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:40.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1614003000 tx=0x7f1614017000 comp rx=0 tx=0).stop 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 shutdown_connections 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> v2:172.21.15.153:6812/83257 conn(0x7f16100367b0 0x7f1610038c60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16240a0e80 0x7f1624142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f162409cc70 0x7f162409b490 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 --2- 172.21.15.12:0/2000230504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f162409fad0 0x7f16241422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 >> 172.21.15.12:0/2000230504 conn(0x7f1624063cc0 msgr2=0x7f162406b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 shutdown_connections 2023-09-29T23:17:40.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:40.254+0000 7f162a3f7700 1 -- 172.21.15.12:0/2000230504 wait complete. 2023-09-29T23:17:43.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.399+0000 7f459ded6700 10 monclient: get_monmap_and_config 2023-09-29T23:17:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.399+0000 7f459ded6700 10 monclient: build_initial_monmap 2023-09-29T23:17:43.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.399+0000 7f459ded6700 10 monclient: monmap: 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:43.400163+0000 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:43.400163+0000 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:43.400 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:43.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 Processor -- start 2023-09-29T23:17:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 -- start start 2023-09-29T23:17:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient: init 2023-09-29T23:17:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:43.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient: _add_conns ranks=[2,3,0,9,6,1,4,5,7,8,10] 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 0x7f459806f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-b con 0x7f459812dbb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459812e710 0x7f459806f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-g con 0x7f459812e710 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45981310e0 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4598068ed0 con 0x7f45981310e0 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459806fad0 con 0x7f459812dbb0 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a0060 con 0x7f459812e710 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:43.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ded6700 10 monclient(hunting): authenticate will time out at 1372.760614s 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459d6d5700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f45981310e0 0x7f45981314f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51658/0 (socket says 172.21.15.12:51658) 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 0x7f459806f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459d6d5700 1 -- 172.21.15.12:0/1193593232 learned_addr learned my addr 172.21.15.12:0/1193593232 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): get_auth_request con 0x7f459812dbb0 auth_method 0 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f4597fff700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459812e710 0x7f459806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f459ced4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f4597fff700 10 monclient(hunting): get_auth_request con 0x7f459812e710 auth_method 0 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f4597fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f4597fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.400+0000 7f4597fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 1 -- 172.21.15.12:0/1193593232 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067922244 0 0) 0x7f4598068ed0 con 0x7f45981310e0 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient(hunting): my global_id is 4986 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 1 -- 172.21.15.12:0/1193593232 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580003570 con 0x7f45981310e0 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:43.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 10 monclient(hunting): handle_auth_done global_id 4637 payload 274 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 -- 172.21.15.12:0/1193593232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459812e710 msgr2=0x7f459806f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459812e710 0x7f459806f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 -- 172.21.15.12:0/1193593232 >> v1:172.21.15.12:6789/0 conn(0x7f45981310e0 legacy=0x7f45981314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 10 monclient: _finish_hunting 0 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 monclient: found mon.noname-b 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 -- 172.21.15.12:0/1193593232 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459804c960 con 0x7f459812dbb0 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ced4700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 0x7f459806f050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4588009910 tx=0x7f45880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe1ff84e5151c8ff server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 1 -- 172.21.15.12:0/1193593232 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4588011ab0 con 0x7f459812dbb0 2023-09-29T23:17:43.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: dump: 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:43.415 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: _finish_auth 0 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:43.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 1 -- 172.21.15.12:0/1193593232 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4588011c30 con 0x7f459812dbb0 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 5 monclient: authenticate success, global_id 4637 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 1 -- 172.21.15.12:0/1193593232 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4588011da0 con 0x7f459812dbb0 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 4 set_mon_vals no callback set 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f45977fe700 10 monclient: dump: 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:43.417 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:43.418 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 10 monclient: shutdown 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 1 -- 172.21.15.12:0/1193593232 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 msgr2=0x7f459806f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.401+0000 7f459ded6700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 0x7f459806f050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4588009910 tx=0x7f45880091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 -- 172.21.15.12:0/1193593232 shutdown_connections 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f459812dbb0 0x7f459806f050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 --2- 172.21.15.12:0/1193593232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f459812e710 0x7f459806f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 -- 172.21.15.12:0/1193593232 >> 172.21.15.12:0/1193593232 conn(0x7f4598063e30 msgr2=0x7f45980a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 -- 172.21.15.12:0/1193593232 shutdown_connections 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 -- 172.21.15.12:0/1193593232 wait complete. 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 10 monclient: build_initial_monmap 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 10 monclient: monmap: 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:43.420 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.402+0000 7f459ded6700 1 Processor -- start 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 -- start start 2023-09-29T23:17:43.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient: init 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient: _add_conns ranks=[1,8,4,6,3,5,2,0,7] 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 0x7f4598140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-f con 0x7f459812dbb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-e con 0x7f459812e710 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f45981310e0 0x7f45980987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): picked mon.noname-c con 0x7f45981310e0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a04b0 con 0x7f459812dbb0 2023-09-29T23:17:43.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f459811a910 con 0x7f45981310e0 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f459812b160 con 0x7f459812e710 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ded6700 10 monclient(hunting): authenticate will time out at 1372.763731s 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f45981310e0 0x7f45980987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:60424/0 (socket says 172.21.15.12:60424) 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 1 -- 172.21.15.12:0/121754891 learned_addr learned my addr 172.21.15.12:0/121754891 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): get_auth_request con 0x7f459812e710 auth_method 0 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:43.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): get_auth_request con 0x7f45981310e0 auth_method 0 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ced4700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 0x7f4598140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459d6d5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ced4700 10 monclient(hunting): get_auth_request con 0x7f459812dbb0 auth_method 0 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.403+0000 7f459ced4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ced4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ced4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 10 monclient(hunting): handle_auth_done global_id 4688 payload 824 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 -- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f45981310e0 msgr2=0x7f45980987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f45981310e0 0x7f45980987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 -- 172.21.15.12:0/121754891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 msgr2=0x7f4598140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 0x7f4598140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 monclient: found mon.noname-e 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 -- 172.21.15.12:0/121754891 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588009420 con 0x7f459812e710 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ced4700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 0x7f4598140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4597fff700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f458c00ec80 tx=0x7f458c0091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f458c00ccd0 con 0x7f459812e710 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:43.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: dump: 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:43.427 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: _finish_auth 0 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 5 monclient: authenticate success, global_id 4688 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c00ce70 con 0x7f459812e710 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 10 monclient: _renew_subs 2023-09-29T23:17:43.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459811ac10 con 0x7f459812e710 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f458c0138f0 con 0x7f459812e710 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4594ff9700 10 monclient: dump: 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:43.429 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:43.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f4596ffd700 4 set_mon_vals no callback set 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 10 monclient: _renew_subs 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.404+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459811b1d0 con 0x7f459812e710 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f458c013470 con 0x7f459812e710 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ded6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ded6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ded6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459806e4a0 con 0x7f459812e710 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f4594ff9700 1 --2- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 0x7f4584038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:43.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f458c0479c0 con 0x7f459812e710 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ced4700 1 --2- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 0x7f4584038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.405+0000 7f459ced4700 10 monclient: get_auth_request con 0x7f4584036850 auth_method 0 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.406+0000 7f459ced4700 1 --2- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 0x7f4584038d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4588000c00 tx=0x7f4588017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.410+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f458c01b3b0 con 0x7f459812e710 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.410+0000 7f4594ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:43.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.410+0000 7f4594ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.647+0000 7f459ded6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.647+0000 7f459ded6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.647+0000 7f459ded6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.647+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4598070bc0 con 0x7f459812e710 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.648+0000 7f4594ff9700 1 -- 172.21.15.12:0/121754891 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f458c011350 con 0x7f459812e710 2023-09-29T23:17:43.648 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.648+0000 7f4594ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:43.649 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.648+0000 7f4594ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:43.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:43.649 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:43.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 msgr2=0x7f4584038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 0x7f4584038d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4588000c00 tx=0x7f4588017000 comp rx=0 tx=0).stop 2023-09-29T23:17:43.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 10 monclient: shutdown 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 msgr2=0x7f4598098280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f458c00ec80 tx=0x7f458c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 shutdown_connections 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> v2:172.21.15.153:6812/83257 conn(0x7f4584036850 0x7f4584038d00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f45981310e0 0x7f45980987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f459812dbb0 0x7f4598140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 --2- 172.21.15.12:0/121754891 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f459812e710 0x7f4598098280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 >> 172.21.15.12:0/121754891 conn(0x7f4598063e30 msgr2=0x7f45980a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 shutdown_connections 2023-09-29T23:17:43.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:43.650+0000 7f459ded6700 1 -- 172.21.15.12:0/121754891 wait complete. 2023-09-29T23:17:46.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 10 monclient: get_monmap_and_config 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 10 monclient: build_initial_monmap 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 10 monclient: monmap: 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:46.792 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:46.793007+0000 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:46.793007+0000 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:46.793 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:46.794 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:46.794 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:46.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 1 Processor -- start 2023-09-29T23:17:46.801 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 1 -- start start 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.792+0000 7fe33e8f2700 10 monclient: init 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient: _add_conns ranks=[4,9,0,8,10,6,7,1,3,5,2] 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-c con 0x7fe33806eff0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe33806f3c0 0x7fe33806f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-a con 0x7fe33806f3c0 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3380969e0 addr v1:172.21.15.12:6789/0 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe338068ed0 con 0x7fe3380969e0 2023-09-29T23:17:46.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3380a0be0 con 0x7fe33806f3c0 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe33806de00 con 0x7fe33806eff0 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33e8f2700 10 monclient(hunting): authenticate will time out at 1376.153440s 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe33806f3c0 0x7fe33806f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33ce8f700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe3380969e0 0x7fe338096e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:51666/0 (socket says 172.21.15.12:51666) 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:57950/0 (socket says 172.21.15.12:57950) 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe33806f3c0 0x7fe33806f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38540/0 (socket says 172.21.15.12:38540) 2023-09-29T23:17:46.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe33ce8f700 1 -- 172.21.15.12:0/1566923648 learned_addr learned my addr 172.21.15.12:0/1566923648 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): get_auth_request con 0x7fe33806eff0 auth_method 0 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): get_auth_request con 0x7fe33806f3c0 auth_method 0 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:46.804 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:46.805 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient(hunting): handle_auth_done global_id 4995 payload 274 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 -- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 msgr2=0x7fe338131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 -- 172.21.15.12:0/1566923648 >> v1:172.21.15.12:6789/0 conn(0x7fe3380969e0 legacy=0x7fe338096e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient: _finish_hunting 0 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 monclient: found mon.noname-a 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe336ffd700 1 -- 172.21.15.12:0/1566923648 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784180980 0 0) 0x7fe338068ed0 con 0x7fe3380969e0 2023-09-29T23:17:46.806 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.793+0000 7fe3377fe700 1 -- 172.21.15.12:0/1566923648 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33804c960 con 0x7fe33806f3c0 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe337fff700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe3377fe700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33806f790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe32800dd10 tx=0x7fe32800d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fe1b49836e3dcc server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 1 -- 172.21.15.12:0/1566923648 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe328012070 con 0x7fe33806f3c0 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: dump: 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:46.807 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:46.808 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: _finish_auth 0 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 1 -- 172.21.15.12:0/1566923648 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32800de70 con 0x7fe33806f3c0 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 5 monclient: authenticate success, global_id 4995 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 1 -- 172.21.15.12:0/1566923648 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe32800d2e0 con 0x7fe33806f3c0 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 4 set_mon_vals no callback set 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:46.809 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe336ffd700 10 monclient: dump: 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:46.810 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 10 monclient: shutdown 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 -- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 msgr2=0x7fe33806f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33806f790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe32800dd10 tx=0x7fe32800d630 comp rx=0 tx=0).stop 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 -- 172.21.15.12:0/1566923648 shutdown_connections 2023-09-29T23:17:46.811 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe33806eff0 0x7fe338131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/1566923648 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33806f790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 -- 172.21.15.12:0/1566923648 >> 172.21.15.12:0/1566923648 conn(0x7fe3380a0ea0 msgr2=0x7fe3380a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 -- 172.21.15.12:0/1566923648 shutdown_connections 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.794+0000 7fe33e8f2700 1 -- 172.21.15.12:0/1566923648 wait complete. 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.795+0000 7fe33e8f2700 10 monclient: build_initial_monmap 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.795+0000 7fe33e8f2700 10 monclient: monmap: 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:46.812 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:46.813 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.795+0000 7fe33e8f2700 1 Processor -- start 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 -- start start 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient: init 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient: _add_conns ranks=[6,0,7,3,1,5,8,4,2] 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-d con 0x7fe33806eff0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-a con 0x7fe33806f3c0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3380969e0 0x7fe33811b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): picked mon.noname-i con 0x7fe3380969e0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:46.814 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380a04b0 con 0x7fe33806f3c0 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe33811bca0 con 0x7fe33806eff0 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380a0160 con 0x7fe3380969e0 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33e8f2700 10 monclient(hunting): authenticate will time out at 1376.156656s 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38550/0 (socket says 172.21.15.12:38550) 2023-09-29T23:17:46.815 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:59654/0 (socket says 172.21.15.12:59654) 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 1 -- 172.21.15.12:0/3511653033 learned_addr learned my addr 172.21.15.12:0/3511653033 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): get_auth_request con 0x7fe33806f3c0 auth_method 0 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): get_auth_request con 0x7fe33806eff0 auth_method 0 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:46.816 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe3377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33ce8f700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3380969e0 0x7fe33811b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33ce8f700 10 monclient(hunting): get_auth_request con 0x7fe3380969e0 auth_method 0 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33ce8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33ce8f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe33ce8f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 -- 172.21.15.12:0/3511653033 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3380969e0 msgr2=0x7fe33811b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:46.817 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3380969e0 0x7fe33811b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 -- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 msgr2=0x7fe33811afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient: _finish_hunting 0 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 monclient: found mon.noname-d 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.796+0000 7fe337fff700 1 -- 172.21.15.12:0/3511653033 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320009420 con 0x7fe33806eff0 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe3377fe700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe337fff700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe32000ea00 tx=0x7fe3200091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe32000ccd0 con 0x7fe33806eff0 2023-09-29T23:17:46.818 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: dump: 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:46.819 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: _finish_auth 0 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32000ce50 con 0x7fe33806eff0 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 5 monclient: authenticate success, global_id 4767 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 10 monclient: _renew_subs 2023-09-29T23:17:46.820 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32800d8d0 con 0x7fe33806eff0 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe3200138f0 con 0x7fe33806eff0 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe3367fc700 4 set_mon_vals no callback set 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe32ffff700 10 monclient: dump: 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.821 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:46.822 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 10 monclient: _renew_subs 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.797+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33811c360 con 0x7fe33806eff0 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe320013470 con 0x7fe33806eff0 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe32ffff700 1 --2- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 0x7fe318038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe3200478e0 con 0x7fe33806eff0 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe33e8f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe33e8f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe33e8f2700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:46.823 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe338070bc0 con 0x7fe33806eff0 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe3377fe700 1 --2- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 0x7fe318038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.798+0000 7fe3377fe700 10 monclient: get_auth_request con 0x7fe318036880 auth_method 0 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.799+0000 7fe3377fe700 1 --2- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 0x7fe318038d30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe33809e1c0 tx=0x7fe32801c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.803+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe32001b3b0 con 0x7fe33806eff0 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.803+0000 7fe32ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:46.824 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:46.803+0000 7fe32ffff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.037+0000 7fe33e8f2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.037+0000 7fe33e8f2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.037+0000 7fe33e8f2700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.037+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe33811d490 con 0x7fe33806eff0 2023-09-29T23:17:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.038+0000 7fe32ffff700 1 -- 172.21.15.12:0/3511653033 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe32002a100 con 0x7fe33806eff0 2023-09-29T23:17:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.038+0000 7fe32ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:47.039 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.038+0000 7fe32ffff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:47.039 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:47.039 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:47.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 msgr2=0x7fe318038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 0x7fe318038d30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe33809e1c0 tx=0x7fe32801c000 comp rx=0 tx=0).stop 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 10 monclient: shutdown 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 msgr2=0x7fe33811aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe32000ea00 tx=0x7fe3200091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 shutdown_connections 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> v2:172.21.15.153:6812/83257 conn(0x7fe318036880 0x7fe318038d30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fe3380969e0 0x7fe33811b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fe33806eff0 0x7fe33811aa70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:47.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 --2- 172.21.15.12:0/3511653033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33806f3c0 0x7fe33811afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:47.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 >> 172.21.15.12:0/3511653033 conn(0x7fe3380a0ea0 msgr2=0x7fe33809a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:47.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 shutdown_connections 2023-09-29T23:17:47.042 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:47.040+0000 7fe33e8f2700 1 -- 172.21.15.12:0/3511653033 wait complete. 2023-09-29T23:17:50.052 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 10 monclient: get_monmap_and_config 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 10 monclient: build_initial_monmap 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 10 monclient: monmap: 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:50.182898+0000 2023-09-29T23:17:50.182 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:50.182898+0000 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:50.183 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:50.184 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:50.184 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 1 Processor -- start 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 1 -- start start 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.182+0000 7f9bf3b52700 10 monclient: init 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient: _add_conns ranks=[1,8,2,0,3,4,7,9,5,6,10] 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9bec0a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-e con 0x7f9bec0a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:17:50.193 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-b con 0x7f9bec138980 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:50.194 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec068ed0 con 0x7f9bec0a1f90 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec09e350 con 0x7f9bec138980 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec09fcf0 con 0x7f9bec0a2360 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf3b52700 10 monclient(hunting): authenticate will time out at 1379.543487s 2023-09-29T23:17:50.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:49194/0 (socket says 172.21.15.12:49194) 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:60430/0 (socket says 172.21.15.12:60430) 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 1 -- 172.21.15.12:0/84958974 learned_addr learned my addr 172.21.15.12:0/84958974 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): get_auth_request con 0x7f9bec138980 auth_method 0 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf18ee700 1 --1- 172.21.15.12:0/84958974 >> v1:172.21.15.153:6789/0 conn(0x7f9bec0a1f90 0x7f9bec09f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:53382/0 (socket says 172.21.15.12:53382) 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): get_auth_request con 0x7f9bec0a2360 auth_method 0 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:50.196 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_done global_id 4697 payload 274 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 -- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 msgr2=0x7f9bec13ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf08ec700 1 -- 172.21.15.12:0/84958974 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016546086 0 0) 0x7f9bec068ed0 con 0x7f9bec0a1f90 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 -- 172.21.15.12:0/84958974 >> v1:172.21.15.153:6789/0 conn(0x7f9bec0a1f90 legacy=0x7f9bec09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient: _finish_hunting 0 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 monclient: found mon.noname-e 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.183+0000 7f9bf10ed700 1 -- 172.21.15.12:0/84958974 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec04c960 con 0x7f9bec0a2360 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf20ef700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf10ed700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009910 tx=0x7f9bdc0091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e58f717dc28dc462 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 1 -- 172.21.15.12:0/84958974 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9bdc00bb00 con 0x7f9bec0a2360 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:17:50.198 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: dump: 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:50.199 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: _finish_auth 0 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 1 -- 172.21.15.12:0/84958974 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc00bc80 con 0x7f9bec0a2360 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 5 monclient: authenticate success, global_id 4697 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:50.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 1 -- 172.21.15.12:0/84958974 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9bdc00bdf0 con 0x7f9bec0a2360 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 4 set_mon_vals no callback set 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf08ec700 10 monclient: dump: 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.201 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:50.202 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 10 monclient: shutdown 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 -- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 msgr2=0x7f9bec0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009910 tx=0x7f9bdc0091a0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 -- 172.21.15.12:0/84958974 shutdown_connections 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9bec138980 0x7f9bec13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/84958974 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9bec0a2360 0x7f9bec0a27b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.184+0000 7f9bf3b52700 1 -- 172.21.15.12:0/84958974 >> 172.21.15.12:0/84958974 conn(0x7f9bec0a3b50 msgr2=0x7f9bec0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 1 -- 172.21.15.12:0/84958974 shutdown_connections 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 1 -- 172.21.15.12:0/84958974 wait complete. 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 10 monclient: build_initial_monmap 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 10 monclient: monmap: 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:50.204 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 1 Processor -- start 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 1 -- start start 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.185+0000 7f9bf3b52700 10 monclient: init 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient: _add_conns ranks=[7,0,6,1,5,8,3,2,4] 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9bec0a1f90 0x7f9bec097a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-i con 0x7f9bec0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 0x7f9bec142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-a con 0x7f9bec0a2360 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:17:50.206 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): picked mon.noname-d con 0x7f9bec138980 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0901f0 con 0x7f9bec0a2360 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec09e510 con 0x7f9bec138980 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec08fc30 con 0x7f9bec0a1f90 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:50.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf3b52700 10 monclient(hunting): authenticate will time out at 1379.546364s 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 0x7f9bec142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 0x7f9bec142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38564/0 (socket says 172.21.15.12:38564) 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:59670/0 (socket says 172.21.15.12:59670) 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 1 -- 172.21.15.12:0/883515173 learned_addr learned my addr 172.21.15.12:0/883515173 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): get_auth_request con 0x7f9bec0a2360 auth_method 0 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:50.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): get_auth_request con 0x7f9bec138980 auth_method 0 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9bec0a1f90 0x7f9bec097a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): get_auth_request con 0x7f9bec0a1f90 auth_method 0 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:50.209 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf10ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf18ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient(hunting): handle_auth_done global_id 4776 payload 824 2023-09-29T23:17:50.210 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 -- 172.21.15.12:0/883515173 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9bec0a1f90 msgr2=0x7f9bec097a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9bec0a1f90 0x7f9bec097a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 -- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 msgr2=0x7f9bec142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 0x7f9bec142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient: _finish_hunting 0 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 monclient: found mon.noname-d 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.186+0000 7f9bf20ef700 1 -- 172.21.15.12:0/883515173 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009ae0 con 0x7f9bec138980 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf20ef700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009e70 tx=0x7f9be400b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9be4018070 con 0x7f9bec138980 2023-09-29T23:17:50.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: dump: 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:50.212 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: _finish_auth 0 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 5 monclient: authenticate success, global_id 4776 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be400f040 con 0x7f9bec138980 2023-09-29T23:17:50.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 10 monclient: _renew_subs 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc009420 con 0x7f9bec138980 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9be40092e0 con 0x7f9bec138980 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be1ffb700 10 monclient: dump: 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:50.214 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:50.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 10 monclient: _renew_subs 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9be3fff700 4 set_mon_vals no callback set 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.187+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec08dc50 con 0x7f9bec138980 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9be4003020 con 0x7f9bec138980 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9be1ffb700 1 --2- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 0x7f9bd0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9be4046fd0 con 0x7f9bec138980 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf3b52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf3b52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf3b52700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec08e070 con 0x7f9bec138980 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf18ee700 1 --2- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 0x7f9bd0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:50.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.188+0000 7f9bf18ee700 10 monclient: get_auth_request con 0x7f9bd0036850 auth_method 0 2023-09-29T23:17:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.189+0000 7f9bf18ee700 1 --2- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 0x7f9bd0038d00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9be8007fd0 tx=0x7f9be8007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.193+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9be401c9b0 con 0x7f9bec138980 2023-09-29T23:17:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.193+0000 7f9be1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:50.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.193+0000 7f9be1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:50.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9bf3b52700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:50.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9bf3b52700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:50.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9bf3b52700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:17:50.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9bec070bc0 con 0x7f9bec138980 2023-09-29T23:17:50.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9be1ffb700 1 -- 172.21.15.12:0/883515173 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9be401c3c0 con 0x7f9bec138980 2023-09-29T23:17:50.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9be1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:50.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.429+0000 7f9be1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:50.431 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:50.431 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:50.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 msgr2=0x7f9bd0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 0x7f9bd0038d00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9be8007fd0 tx=0x7f9be8007d30 comp rx=0 tx=0).stop 2023-09-29T23:17:50.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 10 monclient: shutdown 2023-09-29T23:17:50.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:50.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 msgr2=0x7f9bec08c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:50.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009e70 tx=0x7f9be400b040 comp rx=0 tx=0).stop 2023-09-29T23:17:50.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 shutdown_connections 2023-09-29T23:17:50.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> v2:172.21.15.153:6812/83257 conn(0x7f9bd0036850 0x7f9bd0038d00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9bec138980 0x7f9bec08c750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9bec0a1f90 0x7f9bec097a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 --2- 172.21.15.12:0/883515173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9bec0a2360 0x7f9bec142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:50.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 >> 172.21.15.12:0/883515173 conn(0x7f9bec0a3b50 msgr2=0x7f9bec0a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:50.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 shutdown_connections 2023-09-29T23:17:50.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:50.432+0000 7f9bf3b52700 1 -- 172.21.15.12:0/883515173 wait complete. 2023-09-29T23:17:53.444 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.576+0000 7fad0808e700 10 monclient: get_monmap_and_config 2023-09-29T23:17:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.576+0000 7fad0808e700 10 monclient: build_initial_monmap 2023-09-29T23:17:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.576+0000 7fad0808e700 10 monclient: monmap: 2023-09-29T23:17:53.576 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:53.577050+0000 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:53.577050+0000 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:53.577 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.578 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.576+0000 7fad0808e700 1 Processor -- start 2023-09-29T23:17:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 -- start start 2023-09-29T23:17:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient: init 2023-09-29T23:17:53.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient: _add_conns ranks=[7,6,1,9,5,10,4,2,0,3,8] 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fad0009d190 0x7fad0009dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-i con 0x7fad0009d190 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-d con 0x7fad0009ba80 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fad0009e710 addr v1:172.21.15.153:6789/0 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad00068ed0 con 0x7fad0009e710 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fad0009b380 con 0x7fad0009ba80 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fad0009ece0 con 0x7fad0009d190 2023-09-29T23:17:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad0808e700 10 monclient(hunting): authenticate will time out at 1382.937564s 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35178/0 (socket says 172.21.15.12:35178) 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 1 -- 172.21.15.12:0/2027267565 learned_addr learned my addr 172.21.15.12:0/2027267565 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): get_auth_request con 0x7fad0009ba80 auth_method 0 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.577+0000 7fad05e2a700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fad0009d190 0x7fad0009dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05e2a700 10 monclient(hunting): get_auth_request con 0x7fad0009d190 auth_method 0 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05e2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05e2a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05e2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 10 monclient(hunting): handle_auth_done global_id 4785 payload 274 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 -- 172.21.15.12:0/2027267565 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fad0009d190 msgr2=0x7fad0009dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fad0009d190 0x7fad0009dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 -- 172.21.15.12:0/2027267565 >> v1:172.21.15.153:6789/0 conn(0x7fad0009e710 legacy=0x7fad00139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 10 monclient: _finish_hunting 0 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 monclient: found mon.noname-d 2023-09-29T23:17:53.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 -- 172.21.15.12:0/2027267565 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0004c960 con 0x7fad0009ba80 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad05629700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facf00099e0 tx=0x7facf0009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4d9e017eca71c1ed server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 1 -- 172.21.15.12:0/2027267565 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facf000bca0 con 0x7fad0009ba80 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: dump: 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.590 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:53.591 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: _finish_auth 0 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 1 -- 172.21.15.12:0/2027267565 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf000be20 con 0x7fad0009ba80 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 5 monclient: authenticate success, global_id 4785 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 1 -- 172.21.15.12:0/2027267565 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facf00118a0 con 0x7fad0009ba80 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 4 set_mon_vals no callback set 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:17:53.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad04e28700 10 monclient: dump: 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:53.593 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 10 monclient: shutdown 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 -- 172.21.15.12:0/2027267565 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 msgr2=0x7fad0009e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facf00099e0 tx=0x7facf0009250 comp rx=0 tx=0).stop 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 -- 172.21.15.12:0/2027267565 shutdown_connections 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fad0009d190 0x7fad0009dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 --2- 172.21.15.12:0/2027267565 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fad0009ba80 0x7fad0009e1d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.578+0000 7fad0808e700 1 -- 172.21.15.12:0/2027267565 >> 172.21.15.12:0/2027267565 conn(0x7fad00063e10 msgr2=0x7fad000a32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 1 -- 172.21.15.12:0/2027267565 shutdown_connections 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 1 -- 172.21.15.12:0/2027267565 wait complete. 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 10 monclient: build_initial_monmap 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 10 monclient: monmap: 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:53.595 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:53.596 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 1 Processor -- start 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 1 -- start start 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.579+0000 7fad0808e700 10 monclient: init 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient: _add_conns ranks=[1,3,2,7,5,0,8,6,4] 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad0009ba80 0x7fad000986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-f con 0x7fad0009ba80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:17:53.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad0009d190 0x7fad00132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-g con 0x7fad0009d190 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): picked mon.noname-b con 0x7fad0009e710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0006f240 con 0x7fad0009ba80 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0009f680 con 0x7fad0009e710 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad00120380 con 0x7fad0009d190 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:53.598 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0808e700 10 monclient(hunting): authenticate will time out at 1382.940493s 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad0009ba80 0x7fad000986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:46408/0 (socket says 172.21.15.12:46408) 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 -- 172.21.15.12:0/726124804 learned_addr learned my addr 172.21.15.12:0/726124804 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): get_auth_request con 0x7fad0009e710 auth_method 0 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad0009d190 0x7fad00132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): get_auth_request con 0x7fad0009ba80 auth_method 0 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:53.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): get_auth_request con 0x7fad0009d190 auth_method 0 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05e2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:53.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad05629700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient(hunting): handle_auth_done global_id 4655 payload 824 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 -- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad0009d190 msgr2=0x7fad00132f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad0009d190 0x7fad00132f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 -- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad0009ba80 msgr2=0x7fad000986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad0009ba80 0x7fad000986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient: _finish_hunting 0 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 monclient: found mon.noname-b 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:17:53.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.580+0000 7fad0662b700 1 -- 172.21.15.12:0/726124804 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf00094f0 con 0x7fad0009e710 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0662b700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facf800b4e0 tx=0x7facf8006f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facf800be90 con 0x7fad0009e710 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: dump: 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:53.602 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: _finish_auth 0 2023-09-29T23:17:53.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 5 monclient: authenticate success, global_id 4655 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf800f990 con 0x7fad0009e710 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 10 monclient: _renew_subs 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad00120100 con 0x7fad0009e710 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facf8011400 con 0x7fad0009e710 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf5ffb700 10 monclient: dump: 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:53.604 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:53.605 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 10 monclient: _renew_subs 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7facf7fff700 4 set_mon_vals no callback set 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.581+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad00099230 con 0x7fad0009e710 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7facf800fb40 con 0x7fad0009e710 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad0808e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad0808e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad0808e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad00070bc0 con 0x7fad0009e710 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7facf5ffb700 1 --2- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 0x7face4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:53.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7facf8046d20 con 0x7fad0009e710 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad05e2a700 1 --2- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 0x7face4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.582+0000 7fad05e2a700 10 monclient: get_auth_request con 0x7face4036850 auth_method 0 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.583+0000 7fad05e2a700 1 --2- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 0x7face4038d00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facfc009fd0 tx=0x7facfc009880 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.587+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facf8011560 con 0x7fad0009e710 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.587+0000 7facf5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:53.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.587+0000 7facf5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7fad0808e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7fad0808e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7fad0808e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad001204c0 con 0x7fad0009e710 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7facf5ffb700 1 -- 172.21.15.12:0/726124804 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7facf8019360 con 0x7fad0009e710 2023-09-29T23:17:53.827 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7facf5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.827+0000 7facf5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:53.828 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:53.828 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:53.829 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 msgr2=0x7face4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 0x7face4038d00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facfc009fd0 tx=0x7facfc009880 comp rx=0 tx=0).stop 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 10 monclient: shutdown 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 msgr2=0x7fad0011dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facf800b4e0 tx=0x7facf8006f90 comp rx=0 tx=0).stop 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 shutdown_connections 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad0009ba80 0x7fad000986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> v2:172.21.15.153:6812/83257 conn(0x7face4036850 0x7face4038d00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.830 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fad0009e710 0x7fad0011dd10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 --2- 172.21.15.12:0/726124804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad0009d190 0x7fad00132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 >> 172.21.15.12:0/726124804 conn(0x7fad00063e10 msgr2=0x7fad00136000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 shutdown_connections 2023-09-29T23:17:53.831 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:53.829+0000 7fad0808e700 1 -- 172.21.15.12:0/726124804 wait complete. 2023-09-29T23:17:56.842 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:17:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.973+0000 7f779c936700 10 monclient: get_monmap_and_config 2023-09-29T23:17:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.973+0000 7f779c936700 10 monclient: build_initial_monmap 2023-09-29T23:17:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.973+0000 7f779c936700 10 monclient: monmap: 2023-09-29T23:17:56.973 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:17:56.974291+0000 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:17:56.974291+0000 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:56.974 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:17:56.975 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:56.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 Processor -- start 2023-09-29T23:17:56.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 -- start start 2023-09-29T23:17:56.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient: init 2023-09-29T23:17:56.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:56.984 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient: _add_conns ranks=[7,9,4,0,8,2,5,6,10,1,3] 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779806f3a0 0x7f779806b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): picked mon.noname-i con 0x7f779806f3a0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f779809f840 0x7f779806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): picked mon.noname-a con 0x7f779809f840 addr v2:172.21.15.12:3300/0 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): picked mon.noname-c con 0x7f77980969e0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779806b080 con 0x7f779809f840 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f779809f570 con 0x7f77980969e0 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f779806f770 con 0x7f779806f3a0 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f779c936700 10 monclient(hunting): authenticate will time out at 1386.334730s 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f779809f840 0x7f779806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f779809f840 0x7f779806c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55744/0 (socket says 172.21.15.12:55744) 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:54368/0 (socket says 172.21.15.12:54368) 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 1 -- 172.21.15.12:0/807650043 learned_addr learned my addr 172.21.15.12:0/807650043 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:56.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 10 monclient(hunting): get_auth_request con 0x7f779809f840 auth_method 0 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7795d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 10 monclient(hunting): get_auth_request con 0x7f77980969e0 auth_method 0 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.974+0000 7f7796d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:56.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779659c700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779806f3a0 0x7f779806b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779659c700 10 monclient(hunting): get_auth_request con 0x7f779806f3a0 auth_method 0 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient(hunting): handle_auth_done global_id 5031 payload 274 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 -- 172.21.15.12:0/807650043 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779806f3a0 msgr2=0x7f779806b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779806f3a0 0x7f779806b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 -- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 msgr2=0x7f7798096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:56.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 monclient: found mon.noname-a 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 -- 172.21.15.12:0/807650043 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798068ed0 con 0x7f779809f840 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7796d9d700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f7795d9b700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f779809f840 0x7f779806c5a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f77800099b0 tx=0x7f7780009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd2ea2ba0cfaacc server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 1 -- 172.21.15.12:0/807650043 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7780009ea0 con 0x7f779809f840 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: dump: 2023-09-29T23:17:56.989 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:56.990 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: _finish_auth 0 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779c936700 5 monclient: authenticate success, global_id 5031 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 1 -- 172.21.15.12:0/807650043 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778000bce0 con 0x7f779809f840 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779c936700 10 monclient: get_monmap_and_config success 2023-09-29T23:17:56.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 1 -- 172.21.15.12:0/807650043 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f778000be50 con 0x7f779809f840 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779c936700 4 set_mon_vals no callback set 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.975+0000 7f779559a700 10 monclient: dump: 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:56.992 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 10 monclient: shutdown 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 -- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f779809f840 msgr2=0x7f779806c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f779809f840 0x7f779806c5a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f77800099b0 tx=0x7f7780009250 comp rx=0 tx=0).stop 2023-09-29T23:17:56.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 -- 172.21.15.12:0/807650043 shutdown_connections 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77980969e0 0x7f7798096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779806f3a0 0x7f779806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 --2- 172.21.15.12:0/807650043 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f779809f840 0x7f779806c5a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 -- 172.21.15.12:0/807650043 >> 172.21.15.12:0/807650043 conn(0x7f7798063e80 msgr2=0x7f77980a0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 -- 172.21.15.12:0/807650043 shutdown_connections 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 1 -- 172.21.15.12:0/807650043 wait complete. 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 10 monclient: build_initial_monmap 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.976+0000 7f779c936700 10 monclient: monmap: 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:17:56.994 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:17:56.995 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 Processor -- start 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 -- start start 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient: init 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient: _reopen_session rank -1 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient: _add_conns ranks=[4,6,7,8,5,3,1,2,0] 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): picked mon.noname-c con 0x7f779806f3a0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77980969e0 0x7f779811fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): picked mon.noname-d con 0x7f77980969e0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779809f840 0x7f779811fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): picked mon.noname-i con 0x7f779809f840 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f77980a02a0 con 0x7f779806f3a0 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f779806fa10 con 0x7f77980969e0 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): start opening mon connection 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7798143100 con 0x7f779809f840 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): _renew_subs 2023-09-29T23:17:56.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779c936700 10 monclient(hunting): authenticate will time out at 1386.337855s 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f7795d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77980969e0 0x7f779811fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:54370/0 (socket says 172.21.15.12:54370) 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 1 -- 172.21.15.12:0/3591903023 learned_addr learned my addr 172.21.15.12:0/3591903023 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 10 monclient(hunting): get_auth_request con 0x7f779806f3a0 auth_method 0 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f779659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f7795d9b700 10 monclient(hunting): get_auth_request con 0x7f77980969e0 auth_method 0 2023-09-29T23:17:56.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f7795d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.977+0000 7f7795d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7795d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779809f840 0x7f779811fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): get_auth_request con 0x7f779809f840 auth_method 0 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:56.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7795d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7796d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient(hunting): handle_auth_done global_id 4828 payload 824 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 -- 172.21.15.12:0/3591903023 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779809f840 msgr2=0x7f779811fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779809f840 0x7f779811fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 -- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77980969e0 msgr2=0x7f779811fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77980969e0 0x7f779811fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient: _finish_hunting 0 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 monclient: found mon.noname-c 2023-09-29T23:17:57.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 -- 172.21.15.12:0/3591903023 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788009a20 con 0x7f779806f3a0 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779659c700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f779000d890 tx=0x7f7790008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7790007470 con 0x7f779806f3a0 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: dump: 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:57.001 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:57.002 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7790012a70 con 0x7f779806f3a0 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 5 monclient: authenticate success, global_id 4828 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 10 monclient: _renew_subs 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77800094f0 con 0x7f779806f3a0 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7790012be0 con 0x7f779806f3a0 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:17:57.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f7794d99700 4 set_mon_vals no callback set 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f778e7fc700 10 monclient: dump: 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:17:57.004 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 10 monclient: _renew_subs 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.978+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779804c960 con 0x7f779806f3a0 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f779c936700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:57.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f779c936700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f779c936700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7790012d40 con 0x7f779806f3a0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7798066f20 con 0x7f779806f3a0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f778e7fc700 1 --2- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 0x7f7778038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.979+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7790047820 con 0x7f779806f3a0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.980+0000 7f7795d9b700 1 --2- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 0x7f7778038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.980+0000 7f7795d9b700 10 monclient: get_auth_request con 0x7f77780367b0 auth_method 0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.980+0000 7f7795d9b700 1 --2- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 0x7f7778038c60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f779809fc30 tx=0x7f7780017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.984+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77900127d0 con 0x7f779806f3a0 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.984+0000 7f778e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:17:57.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:56.984+0000 7f778e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.222+0000 7f779c936700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.222+0000 7f779c936700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.222+0000 7f779c936700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.222+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7798121770 con 0x7f779806f3a0 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.222+0000 7f778e7fc700 1 -- 172.21.15.12:0/3591903023 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f779001dbb0 con 0x7f779806f3a0 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.223+0000 7f778e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:17:57.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.223+0000 7f778e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:17:57.224 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:17:57.224 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 msgr2=0x7f7778038c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 0x7f7778038c60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f779809fc30 tx=0x7f7780017000 comp rx=0 tx=0).stop 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 10 monclient: shutdown 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 msgr2=0x7f77981360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f779000d890 tx=0x7f7790008040 comp rx=0 tx=0).stop 2023-09-29T23:17:57.225 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 shutdown_connections 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> v2:172.21.15.153:6812/83257 conn(0x7f77780367b0 0x7f7778038c60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f779809f840 0x7f779811fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f779806f3a0 0x7f77981360f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 --2- 172.21.15.12:0/3591903023 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f77980969e0 0x7f779811fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.224+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 >> 172.21.15.12:0/3591903023 conn(0x7f7798063e80 msgr2=0x7f77980981f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.225+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 shutdown_connections 2023-09-29T23:17:57.226 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:17:57.225+0000 7f779c936700 1 -- 172.21.15.12:0/3591903023 wait complete. 2023-09-29T23:18:00.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.369+0000 7f332c976700 10 monclient: get_monmap_and_config 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.369+0000 7f332c976700 10 monclient: build_initial_monmap 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.369+0000 7f332c976700 10 monclient: monmap: 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:00.370610+0000 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:00.370610+0000 2023-09-29T23:18:00.370 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:00.371 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:00.372 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:00.372 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 1 Processor -- start 2023-09-29T23:18:00.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 1 -- start start 2023-09-29T23:18:00.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 10 monclient: init 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 10 monclient: _add_conns ranks=[10,8,6,5,1,2,4,3,0,9,7] 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f33280a1f90 0x7f332809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 10 monclient(hunting): picked mon.noname-f con 0x7f33280a1f90 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 10 monclient(hunting): picked mon.noname-e con 0x7f33280a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.370+0000 7f332c976700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): picked mon.noname-d con 0x7f3328133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332809e350 con 0x7f33280a1f90 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f332809ef70 con 0x7f3328133740 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f332809da70 con 0x7f33280a2360 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient(hunting): authenticate will time out at 1389.731027s 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35192/0 (socket says 172.21.15.12:35192) 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:46662/0 (socket says 172.21.15.12:46662) 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 -- 172.21.15.12:0/834662518 learned_addr learned my addr 172.21.15.12:0/834662518 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): get_auth_request con 0x7f3328133740 auth_method 0 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): get_auth_request con 0x7f33280a2360 auth_method 0 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332659c700 1 --2- 172.21.15.12:0/834662518 >> v2:172.21.15.153:3300/0 conn(0x7f33280a1f90 0x7f332809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332659c700 10 monclient(hunting): get_auth_request con 0x7f33280a1f90 auth_method 0 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient(hunting): handle_auth_done global_id 4803 payload 274 2023-09-29T23:18:00.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 -- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 msgr2=0x7f33280a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 -- 172.21.15.12:0/834662518 >> v2:172.21.15.153:3300/0 conn(0x7f33280a1f90 msgr2=0x7f332809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 --2- 172.21.15.12:0/834662518 >> v2:172.21.15.153:3300/0 conn(0x7f33280a1f90 0x7f332809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 monclient: found mon.noname-d 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 -- 172.21.15.12:0/834662518 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328068ed0 con 0x7f3328133740 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3325d9b700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f3326d9d700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33200099e0 tx=0x7f3320009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=42723968a8665c76 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 1 -- 172.21.15.12:0/834662518 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f332000bca0 con 0x7f3328133740 2023-09-29T23:18:00.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: dump: 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:00.385 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: _finish_auth 0 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 1 -- 172.21.15.12:0/834662518 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000be20 con 0x7f3328133740 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 5 monclient: authenticate success, global_id 4803 2023-09-29T23:18:00.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 1 -- 172.21.15.12:0/834662518 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33200118a0 con 0x7f3328133740 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332559a700 10 monclient: dump: 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:00.387 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:00.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.371+0000 7f332c976700 4 set_mon_vals no callback set 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 10 monclient: shutdown 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 -- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 msgr2=0x7f3328133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33200099e0 tx=0x7f3320009250 comp rx=0 tx=0).stop 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 -- 172.21.15.12:0/834662518 shutdown_connections 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f3328133740 0x7f3328133be0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 --2- 172.21.15.12:0/834662518 >> v2:172.21.15.153:3300/0 conn(0x7f33280a1f90 0x7f332809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 --2- 172.21.15.12:0/834662518 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f33280a2360 0x7f33280a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 -- 172.21.15.12:0/834662518 >> 172.21.15.12:0/834662518 conn(0x7f33280a3b50 msgr2=0x7f33280a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 -- 172.21.15.12:0/834662518 shutdown_connections 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.372+0000 7f332c976700 1 -- 172.21.15.12:0/834662518 wait complete. 2023-09-29T23:18:00.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient: build_initial_monmap 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient: monmap: 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:00.390 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 Processor -- start 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 -- start start 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient: init 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:00.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient: _add_conns ranks=[1,6,5,0,3,4,7,2,8] 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33280a1f90 0x7f332809c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient(hunting): picked mon.noname-f con 0x7f33280a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient(hunting): picked mon.noname-d con 0x7f33280a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3328133740 0x7f332809cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient(hunting): picked mon.noname-h con 0x7f3328133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332809ffc0 con 0x7f33280a1f90 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.373+0000 7f332c976700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3328130f10 con 0x7f3328133740 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:00.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f332812eec0 con 0x7f33280a2360 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient(hunting): authenticate will time out at 1389.734011s 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:35200/0 (socket says 172.21.15.12:35200) 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 -- 172.21.15.12:0/2800051945 learned_addr learned my addr 172.21.15.12:0/2800051945 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): get_auth_request con 0x7f33280a2360 auth_method 0 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33280a1f90 0x7f332809c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): get_auth_request con 0x7f33280a1f90 auth_method 0 2023-09-29T23:18:00.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3328133740 0x7f332809cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): get_auth_request con 0x7f3328133740 auth_method 0 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:00.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3326d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 -- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3328133740 msgr2=0x7f332809cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3328133740 0x7f332809cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 -- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33280a1f90 msgr2=0x7f332809c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33280a1f90 0x7f332809c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient: _finish_hunting 0 2023-09-29T23:18:00.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 monclient: found mon.noname-d 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 -- 172.21.15.12:0/2800051945 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c009420 con 0x7f33280a2360 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3325d9b700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f331c00ec80 tx=0x7f331c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f331c00ccd0 con 0x7f33280a2360 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: dump: 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:00.396 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:00.397 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c00ce30 con 0x7f33280a2360 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 5 monclient: authenticate success, global_id 4812 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient: _renew_subs 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33200094f0 con 0x7f33280a2360 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f331c0138f0 con 0x7f33280a2360 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:00.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f33167fc700 10 monclient: dump: 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:00.399 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f3324d99700 4 set_mon_vals no callback set 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient: _renew_subs 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.374+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332804c960 con 0x7f33280a2360 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f331c013470 con 0x7f33280a2360 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f332c976700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:00.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f332c976700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f332c976700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3328070bc0 con 0x7f33280a2360 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f33167fc700 1 --2- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 0x7f3308038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.375+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f331c0474d0 con 0x7f33280a2360 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.376+0000 7f332659c700 1 --2- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 0x7f3308038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.376+0000 7f332659c700 10 monclient: get_auth_request con 0x7f3308036880 auth_method 0 2023-09-29T23:18:00.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.376+0000 7f332659c700 1 --2- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 0x7f3308038d30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f33180075a0 tx=0x7f33180097e0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:00.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.380+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f331c013720 con 0x7f33280a2360 2023-09-29T23:18:00.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.380+0000 7f33167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:00.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.380+0000 7f33167fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:00.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f332c976700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:00.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f332c976700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f332c976700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3328131ca0 con 0x7f33280a2360 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f33167fc700 1 -- 172.21.15.12:0/2800051945 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f331c01b3b0 con 0x7f33280a2360 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f33167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.616+0000 7f33167fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:00.617 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:00.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 msgr2=0x7f3308038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 0x7f3308038d30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f33180075a0 tx=0x7f33180097e0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 10 monclient: shutdown 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 msgr2=0x7f332809c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f331c00ec80 tx=0x7f331c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 shutdown_connections 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> v2:172.21.15.153:6812/83257 conn(0x7f3308036880 0x7f3308038d30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f3328133740 0x7f332809cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33280a1f90 0x7f332809c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 --2- 172.21.15.12:0/2800051945 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f33280a2360 0x7f332809c9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 >> 172.21.15.12:0/2800051945 conn(0x7f33280a3b50 msgr2=0x7f332806f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 shutdown_connections 2023-09-29T23:18:00.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:00.618+0000 7f332c976700 1 -- 172.21.15.12:0/2800051945 wait complete. 2023-09-29T23:18:03.631 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 10 monclient: get_monmap_and_config 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 10 monclient: build_initial_monmap 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 10 monclient: monmap: 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:03.762899+0000 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:03.762899+0000 2023-09-29T23:18:03.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:03.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 1 Processor -- start 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 1 -- start start 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.762+0000 7fccfd0de700 10 monclient: init 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient: _add_conns ranks=[1,5,10,6,7,8,2,3,9,0,4] 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fccf80a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:03.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-h con 0x7fccf80a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fccf8138980 0x7fccf813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-f con 0x7fccf8138980 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf8068ed0 con 0x7fccf80a1f90 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf809e350 con 0x7fccf8138980 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fccf809fcf0 con 0x7fccf80a2360 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccfd0de700 10 monclient(hunting): authenticate will time out at 1393.123301s 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fccf8138980 0x7fccf813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf6d9d700 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fccf80a1f90 0x7fccf809f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:47036/0 (socket says 172.21.15.12:47036) 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf6d9d700 1 -- 172.21.15.12:0/1733372195 learned_addr learned my addr 172.21.15.12:0/1733372195 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): get_auth_request con 0x7fccf80a2360 auth_method 0 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): get_auth_request con 0x7fccf8138980 auth_method 0 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666237206 0 0) 0x7fccf8068ed0 con 0x7fccf80a1f90 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf659c700 10 monclient(hunting): my global_id is 4942 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce4003570 con 0x7fccf80a1f90 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient(hunting): handle_auth_done global_id 4748 payload 274 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> v2:172.21.15.153:3300/0 conn(0x7fccf8138980 msgr2=0x7fccf813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:03.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> v2:172.21.15.153:3300/0 conn(0x7fccf8138980 0x7fccf813ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> v1:172.21.15.153:6789/0 conn(0x7fccf80a1f90 legacy=0x7fccf809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 monclient: found mon.noname-h 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.153:3302/0 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf804c960 con 0x7fccf80a2360 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.763+0000 7fccf759e700 1 --2- 172.21.15.12:0/1733372195 >> v2:172.21.15.153:3300/0 conn(0x7fccf8138980 0x7fccf813ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.764+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 msgr2=0x7fccf80a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.764+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 msgr2=0x7fccf80a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.764+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fcce00098e0 tx=0x7fcce00091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:18:03.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.764+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:18:03.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient: get_auth_request con 0x7fccf80a2360 auth_method 0 2023-09-29T23:18:03.973 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 0 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 msgr2=0x7fccf80a27b0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fccf80a2360 0x7fccf80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient: _add_conns ranks=[7,9,4,6,0,2,3,8] 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcce4003960 0x7fcce4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): picked mon.noname-i con 0x7fcce4003960 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 --2- 172.21.15.12:0/1733372195 >> v2:172.21.15.12:3300/0 conn(0x7fcce4004810 0x7fcce4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): picked mon.noname-a con 0x7fcce4004810 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:03.974 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4005210 0x7fcce400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): picked mon.noname-c con 0x7fcce4005210 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce400c1b0 con 0x7fcce4004810 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400c2f0 con 0x7fcce4005210 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400c4b0 con 0x7fcce4003960 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:03.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf659c700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:03.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> v2:172.21.15.12:3300/0 conn(0x7fcce4004810 0x7fcce4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4005210 0x7fcce400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): get_auth_request con 0x7fcce4004810 auth_method 0 2023-09-29T23:18:03.977 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): get_auth_request con 0x7fcce4005210 auth_method 0 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcce4003960 0x7fcce4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): get_auth_request con 0x7fcce4003960 auth_method 0 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.965+0000 7fcceffff700 10 monclient(hunting): handle_auth_done global_id 4748 payload 293 2023-09-29T23:18:03.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcce4003960 msgr2=0x7fcce4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcce4003960 0x7fcce4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4005210 msgr2=0x7fcce400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4005210 0x7fcce400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 monclient: found mon.noname-a 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 -- 172.21.15.12:0/1733372195 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf8068ed0 con 0x7fcce4004810 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf759e700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4005210 0x7fcce400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf759e700 1 -- 172.21.15.12:0/1733372195 reap_dead start 2023-09-29T23:18:03.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fcceffff700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce4004810 0x7fcce4004c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcce00093f0 tx=0x7fccf80a3010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a1b389079f5952 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcce0020070 con 0x7fcce4004810 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: dump: 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:03.981 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.982 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: _finish_auth 0 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 5 monclient: authenticate success, global_id 4748 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0011bd0 con 0x7fcce4004810 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 4 set_mon_vals no callback set 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 1 -- 172.21.15.12:0/1733372195 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcce0011650 con 0x7fcce4004810 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccf659c700 10 monclient: dump: 2023-09-29T23:18:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 10 monclient: shutdown 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 1 -- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce4004810 msgr2=0x7fcce4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.966+0000 7fccfd0de700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce4004810 0x7fcce4004c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcce00093f0 tx=0x7fccf80a3010 comp rx=0 tx=0).stop 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 -- 172.21.15.12:0/1733372195 shutdown_connections 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 --2- 172.21.15.12:0/1733372195 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcce4004810 0x7fcce4004c80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 -- 172.21.15.12:0/1733372195 >> 172.21.15.12:0/1733372195 conn(0x7fccf80a3b50 msgr2=0x7fccf80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 -- 172.21.15.12:0/1733372195 shutdown_connections 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 -- 172.21.15.12:0/1733372195 wait complete. 2023-09-29T23:18:03.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 10 monclient: build_initial_monmap 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 10 monclient: monmap: 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:03.986 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.967+0000 7fccfd0de700 1 Processor -- start 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 -- start start 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient: init 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient: _add_conns ranks=[4,8,0,2,7,1,6,3,5] 2023-09-29T23:18:03.987 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-c con 0x7fcce4004810 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fccf8097f70 0x7fccf808c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-e con 0x7fccf8097f70 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccf808cc70 0x7fccf808d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): picked mon.noname-a con 0x7fccf808cc70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf808fd80 con 0x7fccf808cc70 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8097310 con 0x7fcce4004810 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80903b0 con 0x7fccf8097f70 2023-09-29T23:18:03.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccfd0de700 10 monclient(hunting): authenticate will time out at 1393.328565s 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccf808cc70 0x7fccf808d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42474/0 (socket says 172.21.15.12:42474) 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 -- 172.21.15.12:0/3308444363 learned_addr learned my addr 172.21.15.12:0/3308444363 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fccf8097f70 0x7fccf808c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): get_auth_request con 0x7fcce4004810 auth_method 0 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): get_auth_request con 0x7fccf808cc70 auth_method 0 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): get_auth_request con 0x7fccf8097f70 auth_method 0 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.990 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fcceffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient(hunting): handle_auth_done global_id 4837 payload 824 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 -- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fccf8097f70 msgr2=0x7fccf808c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fccf8097f70 0x7fccf808c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 -- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccf808cc70 msgr2=0x7fccf808d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccf808cc70 0x7fccf808d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 monclient: found mon.noname-c 2023-09-29T23:18:03.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.968+0000 7fccf6d9d700 1 -- 172.21.15.12:0/3308444363 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf808ea80 con 0x7fcce4004810 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccf6d9d700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001700 tx=0x7fcce8007de0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcce80048e0 con 0x7fcce4004810 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: dump: 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 5 monclient: authenticate success, global_id 4837 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce8011a10 con 0x7fcce4004810 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 10 monclient: _renew_subs 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf80a0020 con 0x7fcce4004810 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcce8011bc0 con 0x7fcce4004810 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccf5d9b700 4 set_mon_vals no callback set 2023-09-29T23:18:03.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fcceeffd700 10 monclient: dump: 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:03.995 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 10 monclient: _renew_subs 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.969+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf809e490 con 0x7fcce4004810 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fcce8011d20 con 0x7fcce4004810 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fcceeffd700 1 --2- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fccfd0de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fccfd0de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:03.996 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fccfd0de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcce8047320 con 0x7fcce4004810 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf8070bc0 con 0x7fcce4004810 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fcceffff700 1 --2- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.970+0000 7fcceffff700 10 monclient: get_auth_request con 0x7fccd8036850 auth_method 0 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.971+0000 7fcceffff700 1 --2- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 0x7fccd8038d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcce4003930 tx=0x7fcce001a000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.975+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcce8012db0 con 0x7fcce4004810 2023-09-29T23:18:03.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.975+0000 7fcceeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:03.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:03.975+0000 7fcceeffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fccfd0de700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fccfd0de700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fccfd0de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fccf8066790 con 0x7fcce4004810 2023-09-29T23:18:04.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fcceeffd700 1 -- 172.21.15.12:0/3308444363 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcce8012350 con 0x7fcce4004810 2023-09-29T23:18:04.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fcceeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:04.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.211+0000 7fcceeffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:04.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:04.212 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:04.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 msgr2=0x7fccd8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 0x7fccd8038d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcce4003930 tx=0x7fcce001a000 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 10 monclient: shutdown 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 msgr2=0x7fccf8096dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001700 tx=0x7fcce8007de0 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 shutdown_connections 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> v2:172.21.15.153:6812/83257 conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccf808cc70 0x7fccf808d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcce4004810 0x7fccf8096dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 --2- 172.21.15.12:0/3308444363 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fccf8097f70 0x7fccf808c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:04.214 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 >> 172.21.15.12:0/3308444363 conn(0x7fccf80a3b50 msgr2=0x7fccf806b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 shutdown_connections 2023-09-29T23:18:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:04.213+0000 7fccfd0de700 1 -- 172.21.15.12:0/3308444363 wait complete. 2023-09-29T23:18:07.226 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.356+0000 7facdd9f4700 10 monclient: get_monmap_and_config 2023-09-29T23:18:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.356+0000 7facdd9f4700 10 monclient: build_initial_monmap 2023-09-29T23:18:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.356+0000 7facdd9f4700 10 monclient: monmap: 2023-09-29T23:18:07.356 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:07.357313+0000 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:07.357313+0000 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:07.357 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:07.358 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:07.358 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 Processor -- start 2023-09-29T23:18:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 -- start start 2023-09-29T23:18:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient: init 2023-09-29T23:18:07.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient: _add_conns ranks=[2,8,6,7,4,9,0,5,3,10,1] 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 0x7facd809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-b con 0x7facd80a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-e con 0x7facd80a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-d con 0x7facd8133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd809e350 con 0x7facd80a1f90 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7facd809ef70 con 0x7facd8133740 2023-09-29T23:18:07.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7facd809da70 con 0x7facd80a2360 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facdd9f4700 10 monclient(hunting): authenticate will time out at 1396.717606s 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 0x7facd809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56294/0 (socket says 172.21.15.12:56294) 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 1 -- 172.21.15.12:0/1771400037 learned_addr learned my addr 172.21.15.12:0/1771400037 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:57160/0 (socket says 172.21.15.12:57160) 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): get_auth_request con 0x7facd80a2360 auth_method 0 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 0x7facd809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37052/0 (socket says 172.21.15.12:37052) 2023-09-29T23:18:07.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): get_auth_request con 0x7facd8133740 auth_method 0 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 10 monclient(hunting): get_auth_request con 0x7facd80a1f90 auth_method 0 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.357+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 10 monclient(hunting): handle_auth_done global_id 4821 payload 274 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 -- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 msgr2=0x7facd80a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 -- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 msgr2=0x7facd809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:07.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 0x7facd809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 monclient: found mon.noname-d 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 -- 172.21.15.12:0/1771400037 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8068ed0 con 0x7facd8133740 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd67fc700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd77fe700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facd0009940 tx=0x7facd00091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=715f9917b0fb03d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 1 -- 172.21.15.12:0/1771400037 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facd0011ab0 con 0x7facd8133740 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:07.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: dump: 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:07.372 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 1 -- 172.21.15.12:0/1771400037 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0011c30 con 0x7facd8133740 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facdd9f4700 5 monclient: authenticate success, global_id 4821 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facdd9f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facdd9f4700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 1 -- 172.21.15.12:0/1771400037 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facd0011da0 con 0x7facd8133740 2023-09-29T23:18:07.373 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facdd9f4700 4 set_mon_vals no callback set 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.358+0000 7facd5ffb700 10 monclient: dump: 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:07.374 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 10 monclient: shutdown 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 -- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 msgr2=0x7facd8133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facd0009940 tx=0x7facd00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 -- 172.21.15.12:0/1771400037 shutdown_connections 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7facd8133740 0x7facd8133bb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7facd80a1f90 0x7facd809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1771400037 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7facd80a2360 0x7facd80a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 -- 172.21.15.12:0/1771400037 >> 172.21.15.12:0/1771400037 conn(0x7facd80a3b50 msgr2=0x7facd80a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 -- 172.21.15.12:0/1771400037 shutdown_connections 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 1 -- 172.21.15.12:0/1771400037 wait complete. 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 10 monclient: build_initial_monmap 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.359+0000 7facdd9f4700 10 monclient: monmap: 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:07.376 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:07.377 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 Processor -- start 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 -- start start 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient: init 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient: _add_conns ranks=[7,1,0,5,2,8,4,3,6] 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7facd80a1f90 0x7facd8142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-i con 0x7facd80a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80a2360 0x7facd8132980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-f con 0x7facd80a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): picked mon.noname-a con 0x7facd8133740 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:07.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd8130e90 con 0x7facd8133740 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd8142ed0 con 0x7facd80a2360 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7facd8130bb0 con 0x7facd80a1f90 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facdd9f4700 10 monclient(hunting): authenticate will time out at 1396.720704s 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45248/0 (socket says 172.21.15.12:45248) 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 1 -- 172.21.15.12:0/1300436687 learned_addr learned my addr 172.21.15.12:0/1300436687 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): get_auth_request con 0x7facd8133740 auth_method 0 2023-09-29T23:18:07.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd6ffd700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7facd80a1f90 0x7facd8142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd67fc700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80a2360 0x7facd8132980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd6ffd700 10 monclient(hunting): get_auth_request con 0x7facd80a1f90 auth_method 0 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd67fc700 10 monclient(hunting): get_auth_request con 0x7facd80a2360 auth_method 0 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd67fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:07.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.360+0000 7facd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 10 monclient(hunting): handle_auth_done global_id 5049 payload 824 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 -- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7facd80a1f90 msgr2=0x7facd8142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:07.381 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7facd80a1f90 0x7facd8142010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 -- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80a2360 msgr2=0x7facd8132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80a2360 0x7facd8132980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 monclient: found mon.noname-a 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 -- 172.21.15.12:0/1300436687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00094f0 con 0x7facd8133740 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd77fe700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7facd0013000 tx=0x7facd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facd001f070 con 0x7facd8133740 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:07.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: dump: 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:07.383 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 5 monclient: authenticate success, global_id 5049 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0002f30 con 0x7facd8133740 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7facd001ad10 con 0x7facd8133740 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 10 monclient: _renew_subs 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:07.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd0009420 con 0x7facd8133740 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facd57fa700 4 set_mon_vals no callback set 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facceffd700 10 monclient: dump: 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:07.385 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 10 monclient: _renew_subs 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.361+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd804c960 con 0x7facd8133740 2023-09-29T23:18:07.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7facd0023070 con 0x7facd8133740 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facdd9f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facdd9f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facdd9f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd8070bc0 con 0x7facd8133740 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facceffd700 1 --2- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 0x7facb8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7facd0047e80 con 0x7facd8133740 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facd6ffd700 1 --2- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 0x7facb8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.362+0000 7facd6ffd700 10 monclient: get_auth_request con 0x7facb8036850 auth_method 0 2023-09-29T23:18:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.363+0000 7facd6ffd700 1 --2- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 0x7facb8038d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7facc800f570 tx=0x7facc800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.368+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facd00236e0 con 0x7facd8133740 2023-09-29T23:18:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.368+0000 7facceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.368+0000 7facceffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.604+0000 7facdd9f4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.604+0000 7facdd9f4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.604+0000 7facdd9f4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.604+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7facd8130fd0 con 0x7facd8133740 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.605+0000 7facceffd700 1 -- 172.21.15.12:0/1300436687 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7facd8130fd0 con 0x7facd8133740 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.605+0000 7facceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.605+0000 7facceffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:07.605 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:07.606 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 msgr2=0x7facb8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 0x7facb8038d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7facc800f570 tx=0x7facc800b040 comp rx=0 tx=0).stop 2023-09-29T23:18:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 10 monclient: shutdown 2023-09-29T23:18:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:07.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 msgr2=0x7facd812d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7facd0013000 tx=0x7facd0017000 comp rx=0 tx=0).stop 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 shutdown_connections 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> v2:172.21.15.153:6812/83257 conn(0x7facb8036850 0x7facb8038d00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facd8133740 0x7facd812d450 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7facd80a1f90 0x7facd8142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 --2- 172.21.15.12:0/1300436687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80a2360 0x7facd8132980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 >> 172.21.15.12:0/1300436687 conn(0x7facd80a3b50 msgr2=0x7facd806f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 shutdown_connections 2023-09-29T23:18:07.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:07.607+0000 7facdd9f4700 1 -- 172.21.15.12:0/1300436687 wait complete. 2023-09-29T23:18:10.620 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.750+0000 7fbd2728f700 10 monclient: get_monmap_and_config 2023-09-29T23:18:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.750+0000 7fbd2728f700 10 monclient: build_initial_monmap 2023-09-29T23:18:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.750+0000 7fbd2728f700 10 monclient: monmap: 2023-09-29T23:18:10.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:10.751052+0000 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:10.751052+0000 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:10.751 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.750+0000 7fbd2728f700 1 Processor -- start 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 -- start start 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient: init 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient: _add_conns ranks=[8,1,4,5,7,6,10,0,2,3,9] 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-e con 0x7fbd200a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbd200a2360 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-c con 0x7fbd20138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:10.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd20068ed0 con 0x7fbd200a2360 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2009e350 con 0x7fbd20138980 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2009ef70 con 0x7fbd200a1f90 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2728f700 10 monclient(hunting): authenticate will time out at 1400.111510s 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:42480/0 (socket says 172.21.15.12:42480) 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:56306/0 (socket says 172.21.15.12:56306) 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 1 -- 172.21.15.12:0/1242522556 learned_addr learned my addr 172.21.15.12:0/1242522556 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): get_auth_request con 0x7fbd20138980 auth_method 0 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): get_auth_request con 0x7fbd200a1f90 auth_method 0 2023-09-29T23:18:10.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 10 monclient(hunting): handle_auth_done global_id 4846 payload 274 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 1 -- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 msgr2=0x7fbd2009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:10.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.751+0000 7fbd2582c700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 1 -- 172.21.15.12:0/1242522556 >> v1:172.21.15.153:6789/0 conn(0x7fbd200a2360 legacy=0x7fbd200a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 10 monclient: _finish_hunting 0 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 1 monclient: found mon.noname-c 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 1 -- 172.21.15.12:0/1242522556 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2004c960 con 0x7fbd20138980 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2502b700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2582c700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009f50 tx=0x7fbd18009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9bf10e3cedf9c022 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 1 -- 172.21.15.12:0/1242522556 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd1800e070 con 0x7fbd20138980 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:10.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: dump: 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:10.765 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: _finish_auth 0 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 1 -- 172.21.15.12:0/1242522556 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1800be20 con 0x7fbd20138980 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 5 monclient: authenticate success, global_id 4846 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:10.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 1 -- 172.21.15.12:0/1242522556 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd180092e0 con 0x7fbd20138980 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 4 set_mon_vals no callback set 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd17fff700 10 monclient: dump: 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.767 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:10.768 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 10 monclient: shutdown 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 -- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 msgr2=0x7fbd2013ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009f50 tx=0x7fbd18009840 comp rx=0 tx=0).stop 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 -- 172.21.15.12:0/1242522556 shutdown_connections 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbd20138980 0x7fbd2013ad60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1242522556 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbd200a1f90 0x7fbd2009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.752+0000 7fbd2728f700 1 -- 172.21.15.12:0/1242522556 >> 172.21.15.12:0/1242522556 conn(0x7fbd200a3b50 msgr2=0x7fbd200a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 1 -- 172.21.15.12:0/1242522556 shutdown_connections 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 1 -- 172.21.15.12:0/1242522556 wait complete. 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 10 monclient: build_initial_monmap 2023-09-29T23:18:10.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 10 monclient: monmap: 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:10.770 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 1 Processor -- start 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 1 -- start start 2023-09-29T23:18:10.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.753+0000 7fbd2728f700 10 monclient: init 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient: _add_conns ranks=[7,2,6,4,3,5,0,8,1] 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbd200a1f90 0x7fbd201421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-i con 0x7fbd200a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 0x7fbd20097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-b con 0x7fbd200a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): picked mon.noname-d con 0x7fbd20138980 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:10.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2008e1e0 con 0x7fbd200a2360 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200a0030 con 0x7fbd20138980 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2008fef0 con 0x7fbd200a1f90 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2728f700 10 monclient(hunting): authenticate will time out at 1400.114602s 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 0x7fbd20097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:57164/0 (socket says 172.21.15.12:57164) 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 1 -- 172.21.15.12:0/1610109319 learned_addr learned my addr 172.21.15.12:0/1610109319 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:10.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): get_auth_request con 0x7fbd200a2360 auth_method 0 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): get_auth_request con 0x7fbd20138980 auth_method 0 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbd200a1f90 0x7fbd201421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): get_auth_request con 0x7fbd200a1f90 auth_method 0 2023-09-29T23:18:10.774 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2582c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2482a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.754+0000 7fbd2502b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 10 monclient(hunting): handle_auth_done global_id 4830 payload 824 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 -- 172.21.15.12:0/1610109319 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbd200a1f90 msgr2=0x7fbd201421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbd200a1f90 0x7fbd201421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 -- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 msgr2=0x7fbd20097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 0x7fbd20097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 10 monclient: _finish_hunting 0 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 monclient: found mon.noname-d 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 -- 172.21.15.12:0/1610109319 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009420 con 0x7fbd20138980 2023-09-29T23:18:10.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2482a700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 0x7fbd20097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2582c700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800b6c0 tx=0x7fbd18018000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd18003300 con 0x7fbd20138980 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: dump: 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:10.777 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:10.778 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd180092e0 con 0x7fbd20138980 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 5 monclient: authenticate success, global_id 4830 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 10 monclient: _renew_subs 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18009ae0 con 0x7fbd20138980 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbd18007000 con 0x7fbd20138980 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd177fe700 4 set_mon_vals no callback set 2023-09-29T23:18:10.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd157fa700 10 monclient: dump: 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:10.780 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:10.781 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 10 monclient: _renew_subs 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.755+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd200901d0 con 0x7fbd20138980 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbd18023070 con 0x7fbd20138980 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd2728f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd2728f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd2728f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd20070bc0 con 0x7fbd20138980 2023-09-29T23:18:10.782 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd157fa700 1 --2- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 0x7fbd04038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.756+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbd18048fa0 con 0x7fbd20138980 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.757+0000 7fbd2502b700 1 --2- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 0x7fbd04038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.757+0000 7fbd2502b700 10 monclient: get_auth_request con 0x7fbd04036850 auth_method 0 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.757+0000 7fbd2502b700 1 --2- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 0x7fbd04038d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009ea0 tx=0x7fbd1c0091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.761+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd18007ac0 con 0x7fbd20138980 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.761+0000 7fbd157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.761+0000 7fbd157fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.997+0000 7fbd2728f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd2728f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd2728f700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd20066790 con 0x7fbd20138980 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd157fa700 1 -- 172.21.15.12:0/1610109319 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbd1801ae60 con 0x7fbd20138980 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:10.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:10.998+0000 7fbd157fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:10.999 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:10.999 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:11.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 msgr2=0x7fbd04038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 0x7fbd04038d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009ea0 tx=0x7fbd1c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 10 monclient: shutdown 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 msgr2=0x7fbd2008c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800b6c0 tx=0x7fbd18018000 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 shutdown_connections 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbd20138980 0x7fbd2008c590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbd200a1f90 0x7fbd201421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> v2:172.21.15.153:6812/83257 conn(0x7fbd04036850 0x7fbd04038d00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 --2- 172.21.15.12:0/1610109319 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbd200a2360 0x7fbd20097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:11.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 >> 172.21.15.12:0/1610109319 conn(0x7fbd200a3b50 msgr2=0x7fbd2009a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 shutdown_connections 2023-09-29T23:18:11.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:11.000+0000 7fbd2728f700 1 -- 172.21.15.12:0/1610109319 wait complete. 2023-09-29T23:18:14.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:14.143 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.143+0000 7f964ce38700 10 monclient: get_monmap_and_config 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.143+0000 7f964ce38700 10 monclient: build_initial_monmap 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.143+0000 7f964ce38700 10 monclient: monmap: 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:14.144204+0000 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:14.144204+0000 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:14.144 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:14.145 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:14.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.143+0000 7f964ce38700 1 Processor -- start 2023-09-29T23:18:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 -- start start 2023-09-29T23:18:14.146 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient: init 2023-09-29T23:18:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:14.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient: _add_conns ranks=[6,5,4,9,1,8,0,2,7,3,10] 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 0x7f964809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-d con 0x7f96480a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 0x7f96480a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-h con 0x7f96480a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-c con 0x7f9648133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f964809e350 con 0x7f9648133740 2023-09-29T23:18:14.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f964809ef70 con 0x7f96480a2360 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f964809da70 con 0x7f96480a1f90 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964ce38700 10 monclient(hunting): authenticate will time out at 1403.504632s 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 0x7f964809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 0x7f964809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:39756/0 (socket says 172.21.15.12:39756) 2023-09-29T23:18:14.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:50544/0 (socket says 172.21.15.12:50544) 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 1 -- 172.21.15.12:0/2195413803 learned_addr learned my addr 172.21.15.12:0/2195413803 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): get_auth_request con 0x7f96480a1f90 auth_method 0 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): get_auth_request con 0x7f9648133740 auth_method 0 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.144+0000 7f9645d9b700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 0x7f96480a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9645d9b700 10 monclient(hunting): get_auth_request con 0x7f96480a2360 auth_method 0 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9645d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9645d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9645d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 10 monclient(hunting): handle_auth_done global_id 4855 payload 274 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 -- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 msgr2=0x7f964809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 0x7f964809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 -- 172.21.15.12:0/2195413803 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 msgr2=0x7f96480a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 0x7f96480a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 monclient: found mon.noname-c 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:14.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 -- 172.21.15.12:0/2195413803 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648068ed0 con 0x7f9648133740 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9645d9b700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 0x7f96480a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f9646d9d700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9640009ec0 tx=0x7f9640009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d6c821be7f23f84 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 1 -- 172.21.15.12:0/2195413803 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f964000e070 con 0x7f9648133740 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: dump: 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.161 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:14.162 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: _finish_auth 0 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 1 -- 172.21.15.12:0/2195413803 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964000bbf0 con 0x7f9648133740 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 5 monclient: authenticate success, global_id 4855 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:14.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 1 -- 172.21.15.12:0/2195413803 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f964000bd60 con 0x7f9648133740 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 4 set_mon_vals no callback set 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964559a700 10 monclient: dump: 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.164 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:14.165 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 10 monclient: shutdown 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 1 -- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 msgr2=0x7f9648133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.145+0000 7f964ce38700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9640009ec0 tx=0x7f9640009780 comp rx=0 tx=0).stop 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 -- 172.21.15.12:0/2195413803 shutdown_connections 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f96480a1f90 0x7f964809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f9648133740 0x7f9648133bb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 --2- 172.21.15.12:0/2195413803 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f96480a2360 0x7f96480a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 -- 172.21.15.12:0/2195413803 >> 172.21.15.12:0/2195413803 conn(0x7f96480a3b50 msgr2=0x7f96480a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 -- 172.21.15.12:0/2195413803 shutdown_connections 2023-09-29T23:18:14.166 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 1 -- 172.21.15.12:0/2195413803 wait complete. 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 10 monclient: build_initial_monmap 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.146+0000 7f964ce38700 10 monclient: monmap: 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 Processor -- start 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 -- start start 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient: init 2023-09-29T23:18:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient: _add_conns ranks=[2,1,8,3,6,5,7,4,0] 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-b con 0x7f96480a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96480a2360 0x7f964809c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-f con 0x7f96480a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 0x7f964809cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): picked mon.noname-e con 0x7f9648133740 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648130950 con 0x7f96480a2360 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964812f410 con 0x7f96480a1f90 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f964809e510 con 0x7f9648133740 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964ce38700 10 monclient(hunting): authenticate will time out at 1403.507536s 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 0x7f964809cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41202/0 (socket says 172.21.15.12:41202) 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 1 -- 172.21.15.12:0/1359697256 learned_addr learned my addr 172.21.15.12:0/1359697256 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): get_auth_request con 0x7f96480a1f90 auth_method 0 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): get_auth_request con 0x7f9648133740 auth_method 0 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96480a2360 0x7f964809c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): get_auth_request con 0x7f96480a2360 auth_method 0 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:14.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9645d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient(hunting): handle_auth_done global_id 4742 payload 824 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 -- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 msgr2=0x7f964809c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 -- 172.21.15.12:0/1359697256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96480a2360 msgr2=0x7f964809c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96480a2360 0x7f964809c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 monclient: found mon.noname-e 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f9646d9d700 1 -- 172.21.15.12:0/1359697256 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640009a20 con 0x7f9648133740 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.147+0000 7f964659c700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f9646d9d700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 0x7f964809cf00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96480a7620 tx=0x7f9640019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9640003300 con 0x7f9648133740 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: dump: 2023-09-29T23:18:14.174 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:14.175 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964001b3f0 con 0x7f9648133740 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 5 monclient: authenticate success, global_id 4742 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 10 monclient: _renew_subs 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:14.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964804c960 con 0x7f9648133740 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f964001ba20 con 0x7f9648133740 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f963e7fc700 10 monclient: dump: 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:14.177 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:14.178 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f9644d99700 4 set_mon_vals no callback set 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 10 monclient: _renew_subs 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.148+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964809ffc0 con 0x7f9648133740 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9640003050 con 0x7f9648133740 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964ce38700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964ce38700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964ce38700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648130e70 con 0x7f9648133740 2023-09-29T23:18:14.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f963e7fc700 1 --2- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 0x7f9628038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f96400489b0 con 0x7f9648133740 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964659c700 1 --2- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 0x7f9628038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.149+0000 7f964659c700 10 monclient: get_auth_request con 0x7f9628036780 auth_method 0 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.150+0000 7f964659c700 1 --2- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 0x7f9628038c30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9638011540 tx=0x7f9638009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.154+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9648130e70 con 0x7f9648133740 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.154+0000 7f963e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.154+0000 7f963e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.391+0000 7f964ce38700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.391+0000 7f964ce38700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.391+0000 7f964ce38700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.391+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9648130e70 con 0x7f9648133740 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.392+0000 7f963e7fc700 1 -- 172.21.15.12:0/1359697256 <== mon.8 v2:172.21.15.12:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9648130e70 con 0x7f9648133740 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.392+0000 7f963e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:14.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.392+0000 7f963e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:14.393 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:14.393 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:14.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 msgr2=0x7f9628038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:14.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 0x7f9628038c30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9638011540 tx=0x7f9638009250 comp rx=0 tx=0).stop 2023-09-29T23:18:14.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 10 monclient: shutdown 2023-09-29T23:18:14.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 msgr2=0x7f964809cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 0x7f964809cf00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96480a7620 tx=0x7f9640019000 comp rx=0 tx=0).stop 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 shutdown_connections 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9648133740 0x7f964809cf00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> v2:172.21.15.153:6812/83257 conn(0x7f9628036780 0x7f9628038c30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f96480a1f90 0x7f964809c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 --2- 172.21.15.12:0/1359697256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96480a2360 0x7f964809c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 >> 172.21.15.12:0/1359697256 conn(0x7f96480a3b50 msgr2=0x7f9648135dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:14.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 shutdown_connections 2023-09-29T23:18:14.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:14.394+0000 7f964ce38700 1 -- 172.21.15.12:0/1359697256 wait complete. 2023-09-29T23:18:17.407 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 10 monclient: get_monmap_and_config 2023-09-29T23:18:17.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 10 monclient: build_initial_monmap 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 10 monclient: monmap: 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:17.536978+0000 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:17.536978+0000 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:17.537 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 1 Processor -- start 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 1 -- start start 2023-09-29T23:18:17.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.536+0000 7f153b615700 10 monclient: init 2023-09-29T23:18:17.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:17.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient: _add_conns ranks=[10,2,1,4,8,0,9,7,5,3,6] 2023-09-29T23:18:17.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f153406ece0 0x7f1534133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): picked mon.noname-f con 0x7f153406ece0 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): picked mon.noname-b con 0x7f1534133c80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f153406f5f0 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534068ed0 con 0x7f153406f5f0 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153409deb0 con 0x7f153406ece0 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153409f290 con 0x7f1534133c80 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:17.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f153b615700 10 monclient(hunting): authenticate will time out at 1406.897430s 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41204/0 (socket says 172.21.15.12:41204) 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 -- 172.21.15.12:0/3164878508 learned_addr learned my addr 172.21.15.12:0/3164878508 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): get_auth_request con 0x7f1534133c80 auth_method 0 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 1 --2- 172.21.15.12:0/3164878508 >> v2:172.21.15.153:3300/0 conn(0x7f153406ece0 0x7f1534133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): get_auth_request con 0x7f153406ece0 auth_method 0 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f15393b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient(hunting): handle_auth_done global_id 4691 payload 274 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 -- 172.21.15.12:0/3164878508 >> v2:172.21.15.153:3300/0 conn(0x7f153406ece0 msgr2=0x7f1534133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 --2- 172.21.15.12:0/3164878508 >> v2:172.21.15.153:3300/0 conn(0x7f153406ece0 0x7f1534133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 -- 172.21.15.12:0/3164878508 >> v1:172.21.15.153:6789/0 conn(0x7f153406f5f0 legacy=0x7f15340698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient: _finish_hunting 0 2023-09-29T23:18:17.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 monclient: found mon.noname-b 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.537+0000 7f1538bb0700 1 -- 172.21.15.12:0/3164878508 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153404c960 con 0x7f1534133c80 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f1538bb0700 1 --2- 172.21.15.12:0/3164878508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15240099b0 tx=0x7f1524009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b46ac950cbfc80 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 1 -- 172.21.15.12:0/3164878508 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1524009ea0 con 0x7f1534133c80 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: dump: 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.550 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:17.551 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: _finish_auth 0 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 1 -- 172.21.15.12:0/3164878508 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152400bce0 con 0x7f1534133c80 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 5 monclient: authenticate success, global_id 4691 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 1 -- 172.21.15.12:0/3164878508 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f152400be50 con 0x7f1534133c80 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 4 set_mon_vals no callback set 2023-09-29T23:18:17.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f152bfff700 10 monclient: dump: 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:17.553 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 10 monclient: shutdown 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 -- 172.21.15.12:0/3164878508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 msgr2=0x7f153406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 --2- 172.21.15.12:0/3164878508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15240099b0 tx=0x7f1524009250 comp rx=0 tx=0).stop 2023-09-29T23:18:17.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 -- 172.21.15.12:0/3164878508 shutdown_connections 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 --2- 172.21.15.12:0/3164878508 >> v2:172.21.15.153:3300/0 conn(0x7f153406ece0 0x7f1534133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 --2- 172.21.15.12:0/3164878508 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1534133c80 0x7f153406f0b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 -- 172.21.15.12:0/3164878508 >> 172.21.15.12:0/3164878508 conn(0x7f153406aab0 msgr2=0x7f153406aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 -- 172.21.15.12:0/3164878508 shutdown_connections 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.538+0000 7f153b615700 1 -- 172.21.15.12:0/3164878508 wait complete. 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.539+0000 7f153b615700 10 monclient: build_initial_monmap 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.539+0000 7f153b615700 10 monclient: monmap: 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:17.555 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.539+0000 7f153b615700 1 Processor -- start 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.539+0000 7f153b615700 1 -- start start 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.539+0000 7f153b615700 10 monclient: init 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient: _add_conns ranks=[1,0,7,2,6,4,5,8,3] 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153406ece0 0x7f1534142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): picked mon.noname-f con 0x7f153406ece0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): picked mon.noname-a con 0x7f153406f5f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1534133c80 0x7f153411dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): picked mon.noname-i con 0x7f1534133c80 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:17.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1534134830 con 0x7f153406f5f0 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153409e810 con 0x7f153406ece0 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f15341204b0 con 0x7f1534133c80 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f153b615700 10 monclient(hunting): authenticate will time out at 1406.900566s 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60774/0 (socket says 172.21.15.12:60774) 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 1 -- 172.21.15.12:0/454931772 learned_addr learned my addr 172.21.15.12:0/454931772 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): get_auth_request con 0x7f153406f5f0 auth_method 0 2023-09-29T23:18:17.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f15393b1700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153406ece0 0x7f1534142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1539bb2700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1534133c80 0x7f153411dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f15393b1700 10 monclient(hunting): get_auth_request con 0x7f153406ece0 auth_method 0 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f15393b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f15393b1700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:17.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f15393b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1539bb2700 10 monclient(hunting): get_auth_request con 0x7f1534133c80 auth_method 0 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1539bb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1539bb2700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1539bb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 10 monclient(hunting): handle_auth_done global_id 5058 payload 824 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 1 -- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1534133c80 msgr2=0x7f153411dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.540+0000 7f1538bb0700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1534133c80 0x7f153411dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 1 -- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153406ece0 msgr2=0x7f1534142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153406ece0 0x7f1534142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 10 monclient: _finish_hunting 0 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 1 monclient: found mon.noname-a 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:17.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 1 -- 172.21.15.12:0/454931772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240094f0 con 0x7f153406f5f0 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f1538bb0700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f153409f3d0 tx=0x7f1524017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f152400be50 con 0x7f153406f5f0 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: dump: 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.561 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:17.562 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 5 monclient: authenticate success, global_id 5058 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15240118a0 con 0x7f153406f5f0 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 10 monclient: _renew_subs 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153411fee0 con 0x7f153406f5f0 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1524011e90 con 0x7f153406f5f0 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:17.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f152b7fe700 4 set_mon_vals no callback set 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f15297fa700 10 monclient: dump: 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:17.564 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 10 monclient: _renew_subs 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.541+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1534120790 con 0x7f153406f5f0 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1524011a10 con 0x7f153406f5f0 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f153b615700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f153b615700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:17.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f153b615700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1534070bc0 con 0x7f153406f5f0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f15297fa700 1 --2- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 0x7f1518038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.542+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f15240484b0 con 0x7f153406f5f0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.543+0000 7f15393b1700 1 --2- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 0x7f1518038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.543+0000 7f15393b1700 10 monclient: get_auth_request con 0x7f1518036850 auth_method 0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.543+0000 7f15393b1700 1 --2- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 0x7f1518038d00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1530009d40 tx=0x7f1530008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.547+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f152402a310 con 0x7f153406f5f0 2023-09-29T23:18:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.547+0000 7f15297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.547+0000 7f15297fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:17.783 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f153b615700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f153b615700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f153b615700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f153409f570 con 0x7f153406f5f0 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f15297fa700 1 -- 172.21.15.12:0/454931772 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1524028080 con 0x7f153406f5f0 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f15297fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.783+0000 7f15297fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:17.784 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 msgr2=0x7f1518038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 0x7f1518038d00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1530009d40 tx=0x7f1530008040 comp rx=0 tx=0).stop 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 10 monclient: shutdown 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 msgr2=0x7f1534130080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.785+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f153409f3d0 tx=0x7f1524017000 comp rx=0 tx=0).stop 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 shutdown_connections 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> v2:172.21.15.153:6812/83257 conn(0x7f1518036850 0x7f1518038d00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.786 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f1534133c80 0x7f153411dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f153406ece0 0x7f1534142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 --2- 172.21.15.12:0/454931772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f153406f5f0 0x7f1534130080 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 >> 172.21.15.12:0/454931772 conn(0x7f153406aab0 msgr2=0x7f1534099260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 shutdown_connections 2023-09-29T23:18:17.787 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:17.786+0000 7f153b615700 1 -- 172.21.15.12:0/454931772 wait complete. 2023-09-29T23:18:20.798 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.928+0000 7fcd814a3700 10 monclient: get_monmap_and_config 2023-09-29T23:18:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.928+0000 7fcd814a3700 10 monclient: build_initial_monmap 2023-09-29T23:18:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.928+0000 7fcd814a3700 10 monclient: monmap: 2023-09-29T23:18:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:20.929 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:20.929750+0000 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:20.929750+0000 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:20.930 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 Processor -- start 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 -- start start 2023-09-29T23:18:20.931 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient: init 2023-09-29T23:18:20.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:20.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient: _add_conns ranks=[7,10,2,6,9,1,8,5,4,0,3] 2023-09-29T23:18:20.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c0a1f90 0x7fcd7c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-i con 0x7fcd7c0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:20.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 0x7fcd7c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-f con 0x7fcd7c0a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-b con 0x7fcd7c133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c09e350 con 0x7fcd7c0a2360 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c09ef70 con 0x7fcd7c133740 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c09da70 con 0x7fcd7c0a1f90 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.929+0000 7fcd814a3700 10 monclient(hunting): authenticate will time out at 1410.289997s 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41220/0 (socket says 172.21.15.12:41220) 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 -- 172.21.15.12:0/671515315 learned_addr learned my addr 172.21.15.12:0/671515315 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): get_auth_request con 0x7fcd7c133740 auth_method 0 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 1 --2- 172.21.15.12:0/671515315 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 0x7fcd7c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.941 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c0a1f90 0x7fcd7c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): get_auth_request con 0x7fcd7c0a1f90 auth_method 0 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 10 monclient(hunting): get_auth_request con 0x7fcd7c0a2360 auth_method 0 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:20.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient(hunting): handle_auth_done global_id 4700 payload 274 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 -- 172.21.15.12:0/671515315 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c0a1f90 msgr2=0x7fcd7c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c0a1f90 0x7fcd7c09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 -- 172.21.15.12:0/671515315 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 msgr2=0x7fcd7c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 --2- 172.21.15.12:0/671515315 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 0x7fcd7c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 monclient: found mon.noname-b 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 -- 172.21.15.12:0/671515315 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c068ed0 con 0x7fcd7c133740 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7a7fc700 1 --2- 172.21.15.12:0/671515315 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 0x7fcd7c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd7b7fe700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009f50 tx=0x7fcd74009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94417429595f341d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:20.943 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 1 -- 172.21.15.12:0/671515315 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcd7400e070 con 0x7fcd7c133740 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: dump: 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:20.944 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd814a3700 5 monclient: authenticate success, global_id 4700 2023-09-29T23:18:20.945 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 1 -- 172.21.15.12:0/671515315 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7400be40 con 0x7fcd7c133740 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd814a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd814a3700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd814a3700 4 set_mon_vals no callback set 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 1 -- 172.21.15.12:0/671515315 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcd740092e0 con 0x7fcd7c133740 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.930+0000 7fcd79ffb700 10 monclient: dump: 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.946 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:20.947 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 10 monclient: shutdown 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 -- 172.21.15.12:0/671515315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 msgr2=0x7fcd7c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009f50 tx=0x7fcd74009840 comp rx=0 tx=0).stop 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 -- 172.21.15.12:0/671515315 shutdown_connections 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fcd7c133740 0x7fcd7c133be0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 --2- 172.21.15.12:0/671515315 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c0a1f90 0x7fcd7c09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 --2- 172.21.15.12:0/671515315 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0a2360 0x7fcd7c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 -- 172.21.15.12:0/671515315 >> 172.21.15.12:0/671515315 conn(0x7fcd7c0a3b50 msgr2=0x7fcd7c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 -- 172.21.15.12:0/671515315 shutdown_connections 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.931+0000 7fcd814a3700 1 -- 172.21.15.12:0/671515315 wait complete. 2023-09-29T23:18:20.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient: build_initial_monmap 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient: monmap: 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:20.949 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 Processor -- start 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 -- start start 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient: init 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient: _add_conns ranks=[3,4,7,2,6,5,8,0,1] 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 0x7fcd7c142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.950 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-g con 0x7fcd7c0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-c con 0x7fcd7c0a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 0x7fcd7c12d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): picked mon.noname-i con 0x7fcd7c133740 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c12f7e0 con 0x7fcd7c0a1f90 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c09ffc0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c09e510 con 0x7fcd7c133740 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:20.951 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.932+0000 7fcd814a3700 10 monclient(hunting): authenticate will time out at 1410.293000s 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:50572/0 (socket says 172.21.15.12:50572) 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 -- 172.21.15.12:0/3767318499 learned_addr learned my addr 172.21.15.12:0/3767318499 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): get_auth_request con 0x7fcd7c0a2360 auth_method 0 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 0x7fcd7c142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.952 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 0x7fcd7c12d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 10 monclient(hunting): get_auth_request con 0x7fcd7c0a1f90 auth_method 0 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 10 monclient(hunting): get_auth_request con 0x7fcd7c133740 auth_method 0 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient(hunting): handle_auth_done global_id 4864 payload 824 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 -- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 msgr2=0x7fcd7c12d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 0x7fcd7c12d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.953 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 -- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 msgr2=0x7fcd7c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 0x7fcd7c142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 monclient: found mon.noname-c 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 -- 172.21.15.12:0/3767318499 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd74009ae0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7affd700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 0x7fcd7c142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7b7fe700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 0x7fcd7c12d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd7a7fc700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e920 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcd7000eec0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.954 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: dump: 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:20.955 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:20.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 5 monclient: authenticate success, global_id 4864 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7000ca60 con 0x7fcd7c0a2360 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 10 monclient: _renew_subs 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c04c960 con 0x7fcd7c0a2360 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcd7000cbd0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd62ffd700 10 monclient: dump: 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:20.957 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd797fa700 4 set_mon_vals no callback set 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 10 monclient: _renew_subs 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.933+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c130d20 con 0x7fcd7c0a2360 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fcd7000cd30 con 0x7fcd7c0a2360 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd814a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd814a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd814a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c070bc0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd62ffd700 1 --2- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 0x7fcd68038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.934+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcd700470e0 con 0x7fcd7c0a2360 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.935+0000 7fcd7affd700 1 --2- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 0x7fcd68038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.935+0000 7fcd7affd700 10 monclient: get_auth_request con 0x7fcd68036850 auth_method 0 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.935+0000 7fcd7affd700 1 --2- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 0x7fcd68038d00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009ce0 tx=0x7fcd6c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.939+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd7001a720 con 0x7fcd7c0a2360 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.939+0000 7fcd62ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:20.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:20.939+0000 7fcd62ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:21.175 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.175+0000 7fcd814a3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.175+0000 7fcd814a3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.175+0000 7fcd814a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.175+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd7c12f1f0 con 0x7fcd7c0a2360 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.176+0000 7fcd62ffd700 1 -- 172.21.15.12:0/3767318499 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcd7001a360 con 0x7fcd7c0a2360 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.176+0000 7fcd62ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:21.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.176+0000 7fcd62ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:21.177 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:21.177 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 msgr2=0x7fcd68038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 0x7fcd68038d00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009ce0 tx=0x7fcd6c008040 comp rx=0 tx=0).stop 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 10 monclient: shutdown 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 msgr2=0x7fcd7c132af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.177+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e920 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 shutdown_connections 2023-09-29T23:18:21.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> v2:172.21.15.153:6812/83257 conn(0x7fcd68036850 0x7fcd68038d00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0a1f90 0x7fcd7c142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fcd7c133740 0x7fcd7c12d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 --2- 172.21.15.12:0/3767318499 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fcd7c0a2360 0x7fcd7c132af0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 >> 172.21.15.12:0/3767318499 conn(0x7fcd7c0a3b50 msgr2=0x7fcd7c135cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 shutdown_connections 2023-09-29T23:18:21.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:21.178+0000 7fcd814a3700 1 -- 172.21.15.12:0/3767318499 wait complete. 2023-09-29T23:18:24.189 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 10 monclient: get_monmap_and_config 2023-09-29T23:18:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 10 monclient: build_initial_monmap 2023-09-29T23:18:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 10 monclient: monmap: 2023-09-29T23:18:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:24.318 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:24.319021+0000 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:24.319021+0000 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:24.319 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:24.320 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:24.320 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:24.320 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 1 Processor -- start 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 1 -- start start 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.318+0000 7f912721d700 10 monclient: init 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient: _add_conns ranks=[3,6,4,5,7,0,1,9,2,10,8] 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 0x7f9120131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): picked mon.noname-g con 0x7f9120130fe0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): picked mon.noname-d con 0x7f912006f0b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:24.327 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): picked mon.noname-c con 0x7f91200969e0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91201313b0 con 0x7f9120130fe0 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9120131e00 con 0x7f91200969e0 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f912006d690 con 0x7f912006f0b0 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f912721d700 10 monclient(hunting): authenticate will time out at 1413.679456s 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37746/0 (socket says 172.21.15.12:37746) 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:45010/0 (socket says 172.21.15.12:45010) 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 -- 172.21.15.12:0/2898870472 learned_addr learned my addr 172.21.15.12:0/2898870472 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): get_auth_request con 0x7f91200969e0 auth_method 0 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): get_auth_request con 0x7f912006f0b0 auth_method 0 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:24.329 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f9124fb9700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 0x7f9120131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient(hunting): handle_auth_done global_id 4873 payload 274 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 -- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 msgr2=0x7f912006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 -- 172.21.15.12:0/2898870472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 msgr2=0x7f9120131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 0x7f9120131740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.330 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient: _finish_hunting 0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 monclient: found mon.noname-c 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f91257ba700 1 -- 172.21.15.12:0/2898870472 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120068ed0 con 0x7f91200969e0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f9124fb9700 10 monclient: get_auth_request con 0x7f9120130fe0 auth_method 0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f9124fb9700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 0x7f9120131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.319+0000 7f911ffff700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f91257ba700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9118009f50 tx=0x7f9118009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=5db0c8c40bd7fd01 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 1 -- 172.21.15.12:0/2898870472 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f911800e070 con 0x7f91200969e0 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:24.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: dump: 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:24.332 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 1 -- 172.21.15.12:0/2898870472 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911800be80 con 0x7f91200969e0 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 5 monclient: authenticate success, global_id 4873 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:24.333 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 1 -- 172.21.15.12:0/2898870472 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f91180092e0 con 0x7f91200969e0 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 4 set_mon_vals no callback set 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f911f7fe700 10 monclient: dump: 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:24.334 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 10 monclient: shutdown 2023-09-29T23:18:24.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 -- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 msgr2=0x7f9120096e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9118009f50 tx=0x7f9118009840 comp rx=0 tx=0).stop 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 -- 172.21.15.12:0/2898870472 shutdown_connections 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f91200969e0 0x7f9120096e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9120130fe0 0x7f9120131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 --2- 172.21.15.12:0/2898870472 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f912006f0b0 0x7f912006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 -- 172.21.15.12:0/2898870472 >> 172.21.15.12:0/2898870472 conn(0x7f91200a4430 msgr2=0x7f91200a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 -- 172.21.15.12:0/2898870472 shutdown_connections 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.320+0000 7f912721d700 1 -- 172.21.15.12:0/2898870472 wait complete. 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.321+0000 7f912721d700 10 monclient: build_initial_monmap 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.321+0000 7f912721d700 10 monclient: monmap: 2023-09-29T23:18:24.336 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:24.337 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.321+0000 7f912721d700 1 Processor -- start 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 -- start start 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient: init 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient: _add_conns ranks=[4,0,8,2,5,3,1,6,7] 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): picked mon.noname-c con 0x7f912006f0b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 0x7f9120099b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): picked mon.noname-a con 0x7f91200969e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9120130fe0 0x7f912009a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): picked mon.noname-e con 0x7f9120130fe0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9120120a20 con 0x7f91200969e0 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f91200a0550 con 0x7f912006f0b0 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f91200a0180 con 0x7f9120130fe0 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f912721d700 10 monclient(hunting): authenticate will time out at 1413.682463s 2023-09-29T23:18:24.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37748/0 (socket says 172.21.15.12:37748) 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 0x7f9120099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 -- 172.21.15.12:0/3273644964 learned_addr learned my addr 172.21.15.12:0/3273644964 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): get_auth_request con 0x7f912006f0b0 auth_method 0 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9120130fe0 0x7f912009a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): get_auth_request con 0x7f91200969e0 auth_method 0 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:24.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): get_auth_request con 0x7f9120130fe0 auth_method 0 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:24.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f91257ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient(hunting): handle_auth_done global_id 4882 payload 824 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 -- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9120130fe0 msgr2=0x7f912009a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9120130fe0 0x7f912009a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 -- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 msgr2=0x7f9120099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 0x7f9120099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient: _finish_hunting 0 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 monclient: found mon.noname-c 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f9124fb9700 1 -- 172.21.15.12:0/3273644964 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114009420 con 0x7f912006f0b0 2023-09-29T23:18:24.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.322+0000 7f911ffff700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 0x7f9120099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f9124fb9700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f911000c860 tx=0x7f9110008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9110011a80 con 0x7f912006f0b0 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: dump: 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:24.343 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: _finish_auth 0 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:24.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9110011c00 con 0x7f912006f0b0 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 5 monclient: authenticate success, global_id 4882 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9110011d70 con 0x7f912006f0b0 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 10 monclient: _renew_subs 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118009ae0 con 0x7f912006f0b0 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911effd700 4 set_mon_vals no callback set 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f911cff9700 10 monclient: dump: 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:24.345 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:24.346 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 10 monclient: _renew_subs 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.323+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912004c960 con 0x7f912006f0b0 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9110011600 con 0x7f912006f0b0 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f912721d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f912721d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f912721d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9120070bc0 con 0x7f912006f0b0 2023-09-29T23:18:24.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f911cff9700 1 --2- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 0x7f910c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f91100471b0 con 0x7f912006f0b0 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f911ffff700 1 --2- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 0x7f910c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.324+0000 7f911ffff700 10 monclient: get_auth_request con 0x7f910c036850 auth_method 0 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.325+0000 7f911ffff700 1 --2- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 0x7f910c038d00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9114011560 tx=0x7f91140091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.329+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91100118b0 con 0x7f912006f0b0 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.329+0000 7f911cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:24.348 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.329+0000 7f911cff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f912721d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f912721d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f912721d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f912006d970 con 0x7f912006f0b0 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f911cff9700 1 -- 172.21.15.12:0/3273644964 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9110021680 con 0x7f912006f0b0 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f911cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.565+0000 7f911cff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:24.566 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 msgr2=0x7f910c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 0x7f910c038d00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9114011560 tx=0x7f91140091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 10 monclient: shutdown 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 msgr2=0x7f9120099640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f911000c860 tx=0x7f9110008040 comp rx=0 tx=0).stop 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 shutdown_connections 2023-09-29T23:18:24.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> v2:172.21.15.153:6812/83257 conn(0x7f910c036850 0x7f910c038d00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f9120130fe0 0x7f912009a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f912006f0b0 0x7f9120099640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 --2- 172.21.15.12:0/3273644964 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f91200969e0 0x7f9120099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.567+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 >> 172.21.15.12:0/3273644964 conn(0x7f91200a4430 msgr2=0x7f91200a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.568+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 shutdown_connections 2023-09-29T23:18:24.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:24.568+0000 7f912721d700 1 -- 172.21.15.12:0/3273644964 wait complete. 2023-09-29T23:18:26.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.348 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=7,config=2,health=12,kv=8,logm=71} crc {auth=456592471,config=3140514009,health=1307886691,kv=3388759124,logm=3372981452}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=6,config=2,health=12,kv=8,logm=72} crc {auth=1906434292,config=3140514009,health=1307886691,kv=3388759124,logm=3788583270}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=6,config=2,health=12,kv=7,logm=73} crc {auth=2327468441,config=3140514009,health=1307886691,kv=1227192918,logm=2524409551}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=7,config=2,health=11,kv=7,logm=73} crc {auth=456592471,config=3140514009,health=1521142063,kv=3011280722,logm=875200386}) 2023-09-29T23:18:26.349 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=7,config=2,health=12,kv=7,logm=72} crc {auth=456592471,config=3140514009,health=1307886691,kv=2751415474,logm=2910776198}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=7,config=2,health=12,kv=7,logm=72} crc {auth=456592471,config=3140514009,health=1307886691,kv=1263827321,logm=686127960}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=6,config=2,health=11,kv=8,logm=73} crc {auth=1906434292,config=3140514009,health=391782763,kv=3388759124,logm=2861801626}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=8,logm=73} crc {auth=456592471,config=3140514009,health=3447155392,kv=3388759124,logm=1456391322}) 2023-09-29T23:18:26.350 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.337+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=7,config=2,health=12,kv=8,logm=71} crc {auth=456592471,config=3140514009,health=1307886691,kv=3388759124,logm=1980327839}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230002675}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796378088}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3119473216}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.351 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3992929402}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1715554041}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2756779184}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1343300764}) 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.352 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489780055}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.338+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1625282843}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.339+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104071656}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100048328}) 2023-09-29T23:18:26.353 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2890045247}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1080518318}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1029400794}) 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.354 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1041558933}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=310314017}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11000384}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.340+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2101021676}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2355538058}) 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.355 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567853522}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4271462995}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2700832033}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2574107207}) 2023-09-29T23:18:26.356 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=620608398}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451949624}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257210988}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.341+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1001459060}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3961074145}) 2023-09-29T23:18:26.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3748799709}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2627533814}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2488171498}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.358 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2478700234}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1788093535}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.342+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.343+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.343+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=71582462}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.343+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.343+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624772522}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.343+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900638674}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.359 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2610418267}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1897261195}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=923152144}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2672602690}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=722691199}) 2023-09-29T23:18:26.360 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3313315340}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2794966504}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921862055}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.344+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1117665512}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825426135}) 2023-09-29T23:18:26.361 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3803893129}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1034367204}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4120749588}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.362 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4011839089}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3440699044}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3663046509}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286024403}) 2023-09-29T23:18:26.363 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.345+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4225301973}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=18,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=38} crc {logm=2968191509,mdsmap=3585761405,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=2474616440,monmap=4279817696,osd_metadata=3926966860,osd_pg_creating=2468821820,osdmap=692314297}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=39} crc {logm=318885353,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=2512024160,monmap=4279817696,osd_metadata=3926966860,osd_pg_creating=2468821820,osdmap=4027897389}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=1,osd_metadata=4,osd_pg_creating=1,osdmap=38} crc {logm=3546774056,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=2512024160,monmap=1228700967,osd_metadata=325873593,osd_pg_creating=2468821820,osdmap=3542321403}) 2023-09-29T23:18:26.366 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=411375682,mon_config_key=2512024160,monmap=4279817696,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1816139471}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=34} crc {logm=4120476328,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=4002321476,mon_config_key=1279066050,monmap=289810564,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=2604795640}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=931017398,mdsmap=1531151339,mgr=1456440680,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=2173116843,mon_config_key=1970569717,monmap=4279817696,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=3386339196}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.367 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=38} crc {logm=44659861,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=2173116843,mon_config_key=2512024160,monmap=4279817696,osd_metadata=2695509272,osd_pg_creating=2468821820,osdmap=616950847}) 2023-09-29T23:18:26.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-29T23:18:26.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=37} crc {logm=318885353,mdsmap=1531151339,mgr=867728128,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=4067754318,monmap=3140514009,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=1548424891}) 2023-09-29T23:18:26.368 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-09-29T23:18:26.349+0000 7fc893d33700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=39} crc {logm=318885353,mdsmap=1531151339,mgr=2562037998,mgr_command_descs=3077679017,mgr_metadata=889114565,mgrstat=766709988,mon_config_key=2512024160,monmap=289810564,osd_metadata=1834336482,osd_pg_creating=2468821820,osdmap=4027897389}) 2023-09-29T23:18:27.580 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:27.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.708+0000 7f10699de700 10 monclient: get_monmap_and_config 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.708+0000 7f10699de700 10 monclient: build_initial_monmap 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.708+0000 7f10699de700 10 monclient: monmap: 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:27.709599+0000 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:27.709599+0000 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:27.709 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:27.710 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 Processor -- start 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 -- start start 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient: init 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient: _add_conns ranks=[4,6,10,2,1,7,5,9,8,0,3] 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): picked mon.noname-c con 0x7f1064133130 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 0x7f1064133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.719 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): picked mon.noname-d con 0x7f1064133500 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f106412fc20 0x7f1064130090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): picked mon.noname-f con 0x7f106412fc20 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10640a27a0 con 0x7f106412fc20 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f106409adf0 con 0x7f1064133130 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f106409bd60 con 0x7f1064133500 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.709+0000 7f10699de700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10699de700 10 monclient(hunting): authenticate will time out at 1417.070006s 2023-09-29T23:18:27.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 0x7f1064133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 0x7f1064133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:45024/0 (socket says 172.21.15.12:45024) 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37758/0 (socket says 172.21.15.12:37758) 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 1 -- 172.21.15.12:0/370648660 learned_addr learned my addr 172.21.15.12:0/370648660 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): get_auth_request con 0x7f1064133500 auth_method 0 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): get_auth_request con 0x7f1064133130 auth_method 0 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:27.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f1063fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10691dd700 1 --2- 172.21.15.12:0/370648660 >> v2:172.21.15.153:3300/0 conn(0x7f106412fc20 0x7f1064130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10691dd700 10 monclient(hunting): get_auth_request con 0x7f106412fc20 auth_method 0 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10691dd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10691dd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10691dd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:27.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient(hunting): handle_auth_done global_id 4891 payload 274 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 -- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 msgr2=0x7f1064133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 --2- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 0x7f1064133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 -- 172.21.15.12:0/370648660 >> v2:172.21.15.153:3300/0 conn(0x7f106412fc20 msgr2=0x7f1064130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 --2- 172.21.15.12:0/370648660 >> v2:172.21.15.153:3300/0 conn(0x7f106412fc20 0x7f1064130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 monclient: found mon.noname-c 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 -- 172.21.15.12:0/370648660 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064068ed0 con 0x7f1064133130 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10689dc700 1 --2- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1058009910 tx=0x7f10580091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=769790ce56eb4915 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 1 -- 172.21.15.12:0/370648660 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f105800bac0 con 0x7f1064133130 2023-09-29T23:18:27.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: dump: 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:27.724 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:27.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10699de700 5 monclient: authenticate success, global_id 4891 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10699de700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 1 -- 172.21.15.12:0/370648660 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800bc40 con 0x7f1064133130 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10699de700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10699de700 4 set_mon_vals no callback set 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 1 -- 172.21.15.12:0/370648660 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f105800bdb0 con 0x7f1064133130 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.710+0000 7f10637fe700 10 monclient: dump: 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:27.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:27.727 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 10 monclient: shutdown 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 -- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 msgr2=0x7f10640a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 --2- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1058009910 tx=0x7f10580091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 -- 172.21.15.12:0/370648660 shutdown_connections 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 --2- 172.21.15.12:0/370648660 >> v2:172.21.15.153:3300/0 conn(0x7f106412fc20 0x7f1064130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 --2- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f1064133130 0x7f10640a0e80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 --2- 172.21.15.12:0/370648660 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1064133500 0x7f1064133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 -- 172.21.15.12:0/370648660 >> 172.21.15.12:0/370648660 conn(0x7f10640a6490 msgr2=0x7f10640a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:27.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 -- 172.21.15.12:0/370648660 shutdown_connections 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.711+0000 7f10699de700 1 -- 172.21.15.12:0/370648660 wait complete. 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.712+0000 7f10699de700 10 monclient: build_initial_monmap 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.712+0000 7f10699de700 10 monclient: monmap: 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:27.729 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.712+0000 7f10699de700 1 Processor -- start 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.712+0000 7f10699de700 1 -- start start 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.712+0000 7f10699de700 10 monclient: init 2023-09-29T23:18:27.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient: _add_conns ranks=[4,1,5,0,7,6,3,2,8] 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): picked mon.noname-c con 0x7f106412fc20 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 0x7f1064142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): picked mon.noname-f con 0x7f1064133130 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 0x7f106406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): picked mon.noname-h con 0x7f1064133500 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106411dac0 con 0x7f1064133130 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f106409b3e0 con 0x7f106412fc20 2023-09-29T23:18:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f106411cba0 con 0x7f1064133500 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient(hunting): authenticate will time out at 1417.073234s 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37774/0 (socket says 172.21.15.12:37774) 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 -- 172.21.15.12:0/3445431589 learned_addr learned my addr 172.21.15.12:0/3445431589 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): get_auth_request con 0x7f106412fc20 auth_method 0 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1063fff700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 0x7f1064142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient(hunting): handle_auth_done global_id 4900 payload 824 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10691dd700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 0x7f106406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 -- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 msgr2=0x7f106406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 0x7f106406d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 -- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 msgr2=0x7f1064142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 0x7f1064142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:27.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 monclient: found mon.noname-c 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 -- 172.21.15.12:0/3445431589 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058009420 con 0x7f106412fc20 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1063fff700 10 monclient: get_auth_request con 0x7f1064133130 auth_method 0 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10691dd700 10 monclient: get_auth_request con 0x7f1064133500 auth_method 0 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1063fff700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 0x7f1064142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10691dd700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 0x7f106406d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10689dc700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f106409af30 tx=0x7f1058017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1058026480 con 0x7f106412fc20 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:27.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: dump: 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: _finish_auth 0 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10580265e0 con 0x7f106412fc20 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 5 monclient: authenticate success, global_id 4900 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient: _renew_subs 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106404c960 con 0x7f106412fc20 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1058026bd0 con 0x7f106412fc20 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1062ffd700 4 set_mon_vals no callback set 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.713+0000 7f1060ff9700 10 monclient: dump: 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f10699de700 10 monclient: _renew_subs 2023-09-29T23:18:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f10699de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106411e360 con 0x7f106412fc20 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1058023030 con 0x7f106412fc20 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f1060ff9700 1 --2- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 0x7f1048038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.714+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1058048910 con 0x7f106412fc20 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f10699de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f10699de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f10699de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10640a1200 con 0x7f106412fc20 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f1063fff700 1 --2- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 0x7f1048038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f1063fff700 10 monclient: get_auth_request con 0x7f1048036850 auth_method 0 2023-09-29T23:18:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.715+0000 7f1063fff700 1 --2- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 0x7f1048038d00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1050006d40 tx=0x7f1050008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:27.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.719+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1058015070 con 0x7f106412fc20 2023-09-29T23:18:27.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.719+0000 7f1060ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:27.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.719+0000 7f1060ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f10699de700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f10699de700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f10699de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:27.956 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1064070bc0 con 0x7f106412fc20 2023-09-29T23:18:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f1060ff9700 1 -- 172.21.15.12:0/3445431589 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1058010380 con 0x7f106412fc20 2023-09-29T23:18:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f1060ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.956+0000 7f1060ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:27.958 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:27.958 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 msgr2=0x7f1048038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 0x7f1048038d00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1050006d40 tx=0x7f1050008040 comp rx=0 tx=0).stop 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 10 monclient: shutdown 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 msgr2=0x7f10641426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f106409af30 tx=0x7f1058017000 comp rx=0 tx=0).stop 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 shutdown_connections 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> v2:172.21.15.153:6812/83257 conn(0x7f1048036850 0x7f1048038d00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.959 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1064133500 0x7f106406d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f106412fc20 0x7f10641426b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 --2- 172.21.15.12:0/3445431589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064133130 0x7f1064142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 >> 172.21.15.12:0/3445431589 conn(0x7f10640a6490 msgr2=0x7f1064134a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 shutdown_connections 2023-09-29T23:18:27.960 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:27.958+0000 7f10699de700 1 -- 172.21.15.12:0/3445431589 wait complete. 2023-09-29T23:18:30.971 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:31.100 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.100+0000 7fed32d70700 10 monclient: get_monmap_and_config 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.100+0000 7fed32d70700 10 monclient: build_initial_monmap 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.100+0000 7fed32d70700 10 monclient: monmap: 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:31.101034+0000 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:31.101034+0000 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:31.101 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.100+0000 7fed32d70700 1 Processor -- start 2023-09-29T23:18:31.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 -- start start 2023-09-29T23:18:31.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient: init 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient: _add_conns ranks=[1,2,4,9,0,10,3,5,6,8,7] 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fed2c0a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-b con 0x7fed2c0a2360 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-c con 0x7fed2c138980 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c068ed0 con 0x7fed2c0a1f90 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2c09e350 con 0x7fed2c0a2360 2023-09-29T23:18:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fed2c09fcf0 con 0x7fed2c138980 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed32d70700 10 monclient(hunting): authenticate will time out at 1420.461492s 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:40632/0 (socket says 172.21.15.12:40632) 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:37778/0 (socket says 172.21.15.12:37778) 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 1 -- 172.21.15.12:0/2950450281 learned_addr learned my addr 172.21.15.12:0/2950450281 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): get_auth_request con 0x7fed2c0a2360 auth_method 0 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): get_auth_request con 0x7fed2c138980 auth_method 0 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient(hunting): handle_auth_done global_id 4909 payload 274 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 -- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 msgr2=0x7fed2c0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 -- 172.21.15.12:0/2950450281 >> v1:172.21.15.153:6789/0 conn(0x7fed2c0a1f90 legacy=0x7fed2c09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.101+0000 7fed3130d700 1 monclient: found mon.noname-c 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed3130d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed3130d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed3130d700 1 -- 172.21.15.12:0/2950450281 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c04c960 con 0x7fed2c138980 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2bfff700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed3130d700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e90 tx=0x7fed24009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2b0364e2ab9bc064 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 1 -- 172.21.15.12:0/2950450281 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed2400e070 con 0x7fed2c138980 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: dump: 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:31.116 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:31.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 1 -- 172.21.15.12:0/2950450281 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2400bc30 con 0x7fed2c138980 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 5 monclient: authenticate success, global_id 4909 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 1 -- 172.21.15.12:0/2950450281 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed2400bda0 con 0x7fed2c138980 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 4 set_mon_vals no callback set 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:31.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed2b7fe700 10 monclient: dump: 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 10 monclient: shutdown 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 1 -- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 msgr2=0x7fed2c13ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.102+0000 7fed32d70700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e90 tx=0x7fed24009780 comp rx=0 tx=0).stop 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 -- 172.21.15.12:0/2950450281 shutdown_connections 2023-09-29T23:18:31.120 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fed2c138980 0x7fed2c13ad60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 --2- 172.21.15.12:0/2950450281 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fed2c0a2360 0x7fed2c0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 -- 172.21.15.12:0/2950450281 >> 172.21.15.12:0/2950450281 conn(0x7fed2c0a3b50 msgr2=0x7fed2c0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 -- 172.21.15.12:0/2950450281 shutdown_connections 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 1 -- 172.21.15.12:0/2950450281 wait complete. 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 10 monclient: build_initial_monmap 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.103+0000 7fed32d70700 10 monclient: monmap: 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:31.121 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:31.122 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 Processor -- start 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 -- start start 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient: init 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient: _add_conns ranks=[5,7,3,1,2,0,6,4,8] 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 0x7fed2c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-h con 0x7fed2c0a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-i con 0x7fed2c0a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:31.123 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 0x7fed2c08c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): picked mon.noname-g con 0x7fed2c138980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2c09e490 con 0x7fed2c138980 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fed2c08e2e0 con 0x7fed2c0a1f90 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fed2c0902d0 con 0x7fed2c0a2360 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.104+0000 7fed32d70700 10 monclient(hunting): authenticate will time out at 1420.464796s 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.124 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 0x7fed2c08c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 0x7fed2c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:53916/0 (socket says 172.21.15.12:53916) 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 -- 172.21.15.12:0/4045371239 learned_addr learned my addr 172.21.15.12:0/4045371239 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): get_auth_request con 0x7fed2c0a2360 auth_method 0 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): get_auth_request con 0x7fed2c138980 auth_method 0 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:31.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): get_auth_request con 0x7fed2c0a1f90 auth_method 0 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:31.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 -- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 msgr2=0x7fed2c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 0x7fed2c1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 -- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 msgr2=0x7fed2c08c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 0x7fed2c08c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 monclient: found mon.noname-i 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:18:31.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 -- 172.21.15.12:0/4045371239 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c0094f0 con 0x7fed2c0a2360 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed3130d700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 0x7fed2c08c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed30b0c700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 0x7fed2c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed2bfff700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00ec80 tx=0x7fed1c009250 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.105+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed1c00ccd0 con 0x7fed2c0a2360 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: dump: 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.128 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:31.129 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: _finish_auth 0 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c00ce30 con 0x7fed2c0a2360 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed1c0138f0 con 0x7fed2c0a2360 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 5 monclient: authenticate success, global_id 4696 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _renew_subs 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed24009a20 con 0x7fed2c0a2360 2023-09-29T23:18:31.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed2affd700 4 set_mon_vals no callback set 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 10 monclient: dump: 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:31.131 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _renew_subs 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2c142fb0 con 0x7fed2c0a2360 2023-09-29T23:18:31.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fed1c013470 con 0x7fed2c0a2360 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.106+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2c070bc0 con 0x7fed2c0a2360 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.107+0000 7fed28ff9700 1 --2- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 0x7fed0c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.107+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fed1c047510 con 0x7fed2c0a2360 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.107+0000 7fed30b0c700 1 --2- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 0x7fed0c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.107+0000 7fed30b0c700 10 monclient: get_auth_request con 0x7fed0c036880 auth_method 0 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.107+0000 7fed30b0c700 1 --2- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 0x7fed0c038d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed14001de0 tx=0x7fed14009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.112+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed1c013720 con 0x7fed2c0a2360 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.112+0000 7fed28ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.112+0000 7fed28ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.346+0000 7fed32d70700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:31.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.346+0000 7fed32d70700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.346+0000 7fed32d70700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.346+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fed2c066790 con 0x7fed2c0a2360 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.347+0000 7fed28ff9700 1 -- 172.21.15.12:0/4045371239 <== mon.7 v2:172.21.15.153:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fed1c021020 con 0x7fed2c0a2360 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.347+0000 7fed28ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.347+0000 7fed28ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:31.347 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:31.348 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:31.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 msgr2=0x7fed0c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 0x7fed0c038d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fed14001de0 tx=0x7fed14009040 comp rx=0 tx=0).stop 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 10 monclient: shutdown 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 msgr2=0x7fed2c097b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00ec80 tx=0x7fed1c009250 comp rx=0 tx=0).stop 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 shutdown_connections 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> v2:172.21.15.153:6812/83257 conn(0x7fed0c036880 0x7fed0c038d30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed2c138980 0x7fed2c08c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fed2c0a1f90 0x7fed2c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 --2- 172.21.15.12:0/4045371239 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fed2c0a2360 0x7fed2c097b40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 >> 172.21.15.12:0/4045371239 conn(0x7fed2c0a3b50 msgr2=0x7fed2c0a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 shutdown_connections 2023-09-29T23:18:31.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:31.349+0000 7fed32d70700 1 -- 172.21.15.12:0/4045371239 wait complete. 2023-09-29T23:18:34.361 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:34.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.490+0000 7fe589367700 10 monclient: get_monmap_and_config 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.490+0000 7fe589367700 10 monclient: build_initial_monmap 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.490+0000 7fe589367700 10 monclient: monmap: 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:34.491547+0000 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:34.491547+0000 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:34.491 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:34.492 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 1 Processor -- start 2023-09-29T23:18:34.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 1 -- start start 2023-09-29T23:18:34.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient: init 2023-09-29T23:18:34.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:34.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient: _add_conns ranks=[9,1,3,0,10,6,4,7,8,5,2] 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe58412dbb0 0x7fe58406f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient(hunting): picked mon.noname-a con 0x7fe58412dbb0 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe58412e710 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe58406f590 0x7fe5841317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 10 monclient(hunting): picked mon.noname-g con 0x7fe58406f590 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.491+0000 7fe589367700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe584068ed0 con 0x7fe58412e710 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5840a0060 con 0x7fe58412dbb0 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:34.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58406d840 con 0x7fe58406f590 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 10 monclient(hunting): authenticate will time out at 1423.852019s 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe58412dbb0 0x7fe58406f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7fe58412dbb0 0x7fe58406f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49840/0 (socket says 172.21.15.12:49840) 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 -- 172.21.15.12:0/1349254753 learned_addr learned my addr 172.21.15.12:0/1349254753 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): get_auth_request con 0x7fe58412dbb0 auth_method 0 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:34.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe588b66700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe58406f590 0x7fe5841317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe588b66700 10 monclient(hunting): get_auth_request con 0x7fe58406f590 auth_method 0 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe588b66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe588b66700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe588b66700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient(hunting): handle_auth_done global_id 5076 payload 274 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 -- 172.21.15.12:0/1349254753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe58406f590 msgr2=0x7fe5841317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe58406f590 0x7fe5841317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 -- 172.21.15.12:0/1349254753 >> v1:172.21.15.153:6789/0 conn(0x7fe58412e710 legacy=0x7fe5841310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 monclient: found mon.noname-a 2023-09-29T23:18:34.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 -- 172.21.15.12:0/1349254753 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58404c960 con 0x7fe58412dbb0 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe583fff700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe58412dbb0 0x7fe58406f050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0099b0 tx=0x7fe56c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8ee6055f060f5c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 1 -- 172.21.15.12:0/1349254753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe56c00bce0 con 0x7fe58412dbb0 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: dump: 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.505 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:34.506 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 1 -- 172.21.15.12:0/1349254753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c00be60 con 0x7fe58412dbb0 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 5 monclient: authenticate success, global_id 5076 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 1 -- 172.21.15.12:0/1349254753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe56c0118a0 con 0x7fe58412dbb0 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe589367700 4 set_mon_vals no callback set 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:34.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.492+0000 7fe582ffd700 10 monclient: dump: 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:34.508 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 10 monclient: shutdown 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 -- 172.21.15.12:0/1349254753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe58412dbb0 msgr2=0x7fe58406f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe58412dbb0 0x7fe58406f050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0099b0 tx=0x7fe56c009250 comp rx=0 tx=0).stop 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 -- 172.21.15.12:0/1349254753 shutdown_connections 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe58406f590 0x7fe5841317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 --2- 172.21.15.12:0/1349254753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe58412dbb0 0x7fe58406f050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 -- 172.21.15.12:0/1349254753 >> 172.21.15.12:0/1349254753 conn(0x7fe584063e30 msgr2=0x7fe5840a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 -- 172.21.15.12:0/1349254753 shutdown_connections 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.493+0000 7fe589367700 1 -- 172.21.15.12:0/1349254753 wait complete. 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.494+0000 7fe589367700 10 monclient: build_initial_monmap 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.494+0000 7fe589367700 10 monclient: monmap: 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:34.510 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.494+0000 7fe589367700 1 Processor -- start 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.494+0000 7fe589367700 1 -- start start 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.494+0000 7fe589367700 10 monclient: init 2023-09-29T23:18:34.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient: _add_conns ranks=[4,1,2,3,8,6,7,0,5] 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): picked mon.noname-c con 0x7fe58406f590 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe58412dbb0 0x7fe58409b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): picked mon.noname-f con 0x7fe58412dbb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58412e710 0x7fe58409bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): picked mon.noname-b con 0x7fe58412e710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5840a04b0 con 0x7fe58412dbb0 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:34.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58412e290 con 0x7fe58412e710 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe58412f3d0 con 0x7fe58406f590 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe589367700 10 monclient(hunting): authenticate will time out at 1423.855213s 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:43962/0 (socket says 172.21.15.12:43962) 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 -- 172.21.15.12:0/3978287184 learned_addr learned my addr 172.21.15.12:0/3978287184 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): get_auth_request con 0x7fe58406f590 auth_method 0 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:34.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58412e710 0x7fe58409bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): get_auth_request con 0x7fe58412e710 auth_method 0 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe58412dbb0 0x7fe58409b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): get_auth_request con 0x7fe58412dbb0 auth_method 0 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:34.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe588b66700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe5837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient(hunting): handle_auth_done global_id 4918 payload 824 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 -- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58412e710 msgr2=0x7fe58409bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58412e710 0x7fe58409bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 -- 172.21.15.12:0/3978287184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe58412dbb0 msgr2=0x7fe58409b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe58412dbb0 0x7fe58409b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 monclient: found mon.noname-c 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 -- 172.21.15.12:0/3978287184 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0094f0 con 0x7fe58406f590 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe583fff700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe56c006fd0 tx=0x7fe56c017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe56c006b30 con 0x7fe58406f590 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe57bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe57bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.495+0000 7fe57bfff700 10 monclient: dump: 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:34.516 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:34.517 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 10 monclient: _finish_auth 0 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 5 monclient: authenticate success, global_id 4918 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c01bca0 con 0x7fe58406f590 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 10 monclient: _renew_subs 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58412f990 con 0x7fe58406f590 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:34.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe56c01be10 con 0x7fe58406f590 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe5827fc700 4 set_mon_vals no callback set 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 10 monclient: dump: 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:34.519 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 10 monclient: _renew_subs 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58409cf80 con 0x7fe58406f590 2023-09-29T23:18:34.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.496+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe56c023070 con 0x7fe58406f590 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe589367700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe589367700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe589367700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe57bfff700 1 --2- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 0x7fe564038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584070bc0 con 0x7fe58406f590 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe56c048260 con 0x7fe58406f590 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe5837fe700 1 --2- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 0x7fe564038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe5837fe700 10 monclient: get_auth_request con 0x7fe564036850 auth_method 0 2023-09-29T23:18:34.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.497+0000 7fe5837fe700 1 --2- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 0x7fe564038d00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe58409cea0 tx=0x7fe574009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.501+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe56c023910 con 0x7fe58406f590 2023-09-29T23:18:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.501+0000 7fe57bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.501+0000 7fe57bfff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe589367700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe589367700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe589367700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:34.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe58409dbe0 con 0x7fe58406f590 2023-09-29T23:18:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe57bfff700 1 -- 172.21.15.12:0/3978287184 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe56c02a030 con 0x7fe58406f590 2023-09-29T23:18:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe57bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:34.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.737+0000 7fe57bfff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:34.739 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:34.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 msgr2=0x7fe564038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 0x7fe564038d00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe58409cea0 tx=0x7fe574009040 comp rx=0 tx=0).stop 2023-09-29T23:18:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 10 monclient: shutdown 2023-09-29T23:18:34.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 msgr2=0x7fe58412b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe56c006fd0 tx=0x7fe56c017000 comp rx=0 tx=0).stop 2023-09-29T23:18:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 shutdown_connections 2023-09-29T23:18:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> v2:172.21.15.153:6812/83257 conn(0x7fe564036850 0x7fe564038d00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe58412e710 0x7fe58409bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fe58406f590 0x7fe58412b1c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 --2- 172.21.15.12:0/3978287184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe58412dbb0 0x7fe58409b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 >> 172.21.15.12:0/3978287184 conn(0x7fe584063e30 msgr2=0x7fe5840a2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 shutdown_connections 2023-09-29T23:18:34.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:34.739+0000 7fe589367700 1 -- 172.21.15.12:0/3978287184 wait complete. 2023-09-29T23:18:37.752 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 10 monclient: get_monmap_and_config 2023-09-29T23:18:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 10 monclient: build_initial_monmap 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 10 monclient: monmap: 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:37.882139+0000 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:37.882139+0000 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:37.882 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 1 Processor -- start 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 1 -- start start 2023-09-29T23:18:37.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.881+0000 7f190b39e700 10 monclient: init 2023-09-29T23:18:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient: _add_conns ranks=[10,8,6,2,1,3,4,5,0,9,7] 2023-09-29T23:18:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1904133130 0x7f19040a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-f con 0x7f1904133130 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:37.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-e con 0x7f1904133500 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-d con 0x7f190412fc20 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19040a27a0 con 0x7f1904133130 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f190409adf0 con 0x7f190412fc20 2023-09-29T23:18:37.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f190409bd60 con 0x7f1904133500 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190b39e700 10 monclient(hunting): authenticate will time out at 1427.242606s 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:52942/0 (socket says 172.21.15.12:52942) 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:35626/0 (socket says 172.21.15.12:35626) 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 1 -- 172.21.15.12:0/338164956 learned_addr learned my addr 172.21.15.12:0/338164956 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): get_auth_request con 0x7f190412fc20 auth_method 0 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): get_auth_request con 0x7f1904133500 auth_method 0 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 1 --2- 172.21.15.12:0/338164956 >> v2:172.21.15.153:3300/0 conn(0x7f1904133130 0x7f19040a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): get_auth_request con 0x7f1904133130 auth_method 0 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.882+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 10 monclient(hunting): handle_auth_done global_id 4866 payload 274 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 -- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 msgr2=0x7f1904133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 -- 172.21.15.12:0/338164956 >> v2:172.21.15.153:3300/0 conn(0x7f1904133130 msgr2=0x7f19040a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 --2- 172.21.15.12:0/338164956 >> v2:172.21.15.153:3300/0 conn(0x7f1904133130 0x7f19040a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 monclient: found mon.noname-d 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 -- 172.21.15.12:0/338164956 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1904068ed0 con 0x7f190412fc20 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f1909b9b700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190ab9d700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0099e0 tx=0x7f18fc009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c1d8e1ee8451b1c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 1 -- 172.21.15.12:0/338164956 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f18fc011ab0 con 0x7f190412fc20 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: dump: 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:37.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: _finish_auth 0 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 5 monclient: authenticate success, global_id 4866 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 1 -- 172.21.15.12:0/338164956 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc011c30 con 0x7f190412fc20 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 1 -- 172.21.15.12:0/338164956 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f18fc011da0 con 0x7f190412fc20 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 4 set_mon_vals no callback set 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190939a700 10 monclient: dump: 2023-09-29T23:18:37.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:37.901 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 10 monclient: shutdown 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 -- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 msgr2=0x7f19041300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0099e0 tx=0x7f18fc009250 comp rx=0 tx=0).stop 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 -- 172.21.15.12:0/338164956 shutdown_connections 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f190412fc20 0x7f19041300c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 --2- 172.21.15.12:0/338164956 >> v2:172.21.15.153:3300/0 conn(0x7f1904133130 0x7f19040a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 --2- 172.21.15.12:0/338164956 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f1904133500 0x7f1904133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.883+0000 7f190b39e700 1 -- 172.21.15.12:0/338164956 >> 172.21.15.12:0/338164956 conn(0x7f19040a6490 msgr2=0x7f19040a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 1 -- 172.21.15.12:0/338164956 shutdown_connections 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 1 -- 172.21.15.12:0/338164956 wait complete. 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 10 monclient: build_initial_monmap 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 10 monclient: monmap: 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:37.903 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 1 Processor -- start 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 1 -- start start 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.884+0000 7f190b39e700 10 monclient: init 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient: _add_conns ranks=[3,2,6,1,7,5,0,4,8] 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f190412fc20 0x7f190406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-g con 0x7f190412fc20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-b con 0x7f1904133130 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): picked mon.noname-d con 0x7f1904133500 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190409b020 con 0x7f1904133130 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190411dd00 con 0x7f190412fc20 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f190411e2c0 con 0x7f1904133500 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190b39e700 10 monclient(hunting): authenticate will time out at 1427.245346s 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:52946/0 (socket says 172.21.15.12:52946) 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41284/0 (socket says 172.21.15.12:41284) 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 -- 172.21.15.12:0/1622257742 learned_addr learned my addr 172.21.15.12:0/1622257742 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): get_auth_request con 0x7f1904133500 auth_method 0 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): get_auth_request con 0x7f1904133130 auth_method 0 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f190412fc20 0x7f190406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): get_auth_request con 0x7f190412fc20 auth_method 0 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:37.908 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190a39c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient(hunting): handle_auth_done global_id 4875 payload 824 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 -- 172.21.15.12:0/1622257742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f190412fc20 msgr2=0x7f190406d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f190412fc20 0x7f190406d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 -- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 msgr2=0x7f190406d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 monclient: found mon.noname-d 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:37.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f190ab9d700 1 -- 172.21.15.12:0/1622257742 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f8009420 con 0x7f1904133500 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.885+0000 7f1909b9b700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190ab9d700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006fd0 tx=0x7f18fc017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f18fc006c00 con 0x7f1904133500 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: dump: 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:37.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc0233e0 con 0x7f1904133500 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 5 monclient: authenticate success, global_id 4875 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 10 monclient: _renew_subs 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc0094f0 con 0x7f1904133500 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f18fc023a10 con 0x7f1904133500 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f19027fc700 10 monclient: dump: 2023-09-29T23:18:37.912 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:37.913 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 10 monclient: _renew_subs 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190404c960 con 0x7f1904133500 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.886+0000 7f1908b99700 4 set_mon_vals no callback set 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f18fc023590 con 0x7f1904133500 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190b39e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190b39e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:37.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190b39e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904130200 con 0x7f1904133500 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f19027fc700 1 --2- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 0x7f18f0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f18fc048950 con 0x7f1904133500 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190a39c700 1 --2- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 0x7f18f0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.887+0000 7f190a39c700 10 monclient: get_auth_request con 0x7f18f0036850 auth_method 0 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.888+0000 7f190a39c700 1 --2- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 0x7f18f0038d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f18f400f570 tx=0x7f18f400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.892+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18fc023840 con 0x7f1904133500 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.892+0000 7f19027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:37.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:37.892+0000 7f19027fc700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:38.126 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.126+0000 7f190b39e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.126+0000 7f190b39e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.126+0000 7f190b39e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.126+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1904070bc0 con 0x7f1904133500 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.127+0000 7f19027fc700 1 -- 172.21.15.12:0/1622257742 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f18fc015070 con 0x7f1904133500 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.127+0000 7f19027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:38.127 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.127+0000 7f19027fc700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:38.128 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:38.128 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 msgr2=0x7f18f0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 0x7f18f0038d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f18f400f570 tx=0x7f18f400b040 comp rx=0 tx=0).stop 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 10 monclient: shutdown 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 msgr2=0x7f190406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:38.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006fd0 tx=0x7f18fc017000 comp rx=0 tx=0).stop 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 shutdown_connections 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1904133500 0x7f190406ded0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> v2:172.21.15.153:6812/83257 conn(0x7f18f0036850 0x7f18f0038d00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f190412fc20 0x7f190406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 --2- 172.21.15.12:0/1622257742 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1904133130 0x7f190406d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 >> 172.21.15.12:0/1622257742 conn(0x7f19040a6490 msgr2=0x7f1904130450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 shutdown_connections 2023-09-29T23:18:38.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:38.129+0000 7f190b39e700 1 -- 172.21.15.12:0/1622257742 wait complete. 2023-09-29T23:18:41.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.263+0000 7f268b25c700 10 monclient: get_monmap_and_config 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.263+0000 7f268b25c700 10 monclient: build_initial_monmap 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.263+0000 7f268b25c700 10 monclient: monmap: 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:41.264452+0000 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:41.264452+0000 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:41.264 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:41.265 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:41.266 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 Processor -- start 2023-09-29T23:18:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 -- start start 2023-09-29T23:18:41.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient: init 2023-09-29T23:18:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient: _add_conns ranks=[7,8,5,0,3,4,2,1,9,6,10] 2023-09-29T23:18:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f26840a0220 0x7f268409e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-i con 0x7f26840a0220 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:41.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-e con 0x7f268409fb50 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0f80 0x7f26840a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-h con 0x7f26840a0f80 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f268409db90 con 0x7f26840a0f80 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f268409f170 con 0x7f26840a0220 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f268409dd20 con 0x7f268409fb50 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f268b25c700 10 monclient(hunting): authenticate will time out at 1430.624839s 2023-09-29T23:18:41.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.264+0000 7f2683fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:35632/0 (socket says 172.21.15.12:35632) 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 -- 172.21.15.12:0/1571266469 learned_addr learned my addr 172.21.15.12:0/1571266469 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): get_auth_request con 0x7f268409fb50 auth_method 0 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2688ff8700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f26840a0220 0x7f268409e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26897f9700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0f80 0x7f26840a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26897f9700 10 monclient(hunting): get_auth_request con 0x7f26840a0f80 auth_method 0 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26897f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26897f9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26897f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2688ff8700 10 monclient(hunting): get_auth_request con 0x7f26840a0220 auth_method 0 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2688ff8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2688ff8700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2688ff8700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 -- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f26840a0220 msgr2=0x7f268409e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f26840a0220 0x7f268409e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 -- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0f80 msgr2=0x7f26840a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:41.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0f80 0x7f26840a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 monclient: found mon.noname-e 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 -- 172.21.15.12:0/1571266469 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684068ed0 con 0x7f268409fb50 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f2683fff700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26700099e0 tx=0x7f2670009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=115a3becb0bfab50 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 1 -- 172.21.15.12:0/1571266469 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f267000bce0 con 0x7f268409fb50 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: dump: 2023-09-29T23:18:41.278 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f268b25c700 5 monclient: authenticate success, global_id 4769 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f268b25c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 1 -- 172.21.15.12:0/1571266469 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267000be60 con 0x7f268409fb50 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f268b25c700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f268b25c700 4 set_mon_vals no callback set 2023-09-29T23:18:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 1 -- 172.21.15.12:0/1571266469 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f26700118a0 con 0x7f268409fb50 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.265+0000 7f26837fe700 10 monclient: dump: 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:41.281 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 10 monclient: shutdown 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 -- 172.21.15.12:0/1571266469 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 msgr2=0x7f26840a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26700099e0 tx=0x7f2670009250 comp rx=0 tx=0).stop 2023-09-29T23:18:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 -- 172.21.15.12:0/1571266469 shutdown_connections 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0f80 0x7f26840a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f26840a0220 0x7f268409e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 --2- 172.21.15.12:0/1571266469 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f268409fb50 0x7f26840a05f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 -- 172.21.15.12:0/1571266469 >> 172.21.15.12:0/1571266469 conn(0x7f26840a3080 msgr2=0x7f26840a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 -- 172.21.15.12:0/1571266469 shutdown_connections 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 1 -- 172.21.15.12:0/1571266469 wait complete. 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 10 monclient: build_initial_monmap 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.266+0000 7f268b25c700 10 monclient: monmap: 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:41.284 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 Processor -- start 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 -- start start 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient: init 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient: _add_conns ranks=[7,5,6,8,2,4,0,1,3] 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f268409fb50 0x7f26841472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-i con 0x7f268409fb50 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0220 0x7f268413a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-h con 0x7f26840a0220 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): picked mon.noname-d con 0x7f26840a0f80 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f268406f690 con 0x7f26840a0220 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f268406f1c0 con 0x7f26840a0f80 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f26841273d0 con 0x7f268409fb50 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f268b25c700 10 monclient(hunting): authenticate will time out at 1430.627812s 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:52956/0 (socket says 172.21.15.12:52956) 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 1 -- 172.21.15.12:0/2735378640 learned_addr learned my addr 172.21.15.12:0/2735378640 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 10 monclient(hunting): get_auth_request con 0x7f26840a0f80 auth_method 0 2023-09-29T23:18:41.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f26897f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f2688ff8700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f268409fb50 0x7f26841472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.267+0000 7f2683fff700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0220 0x7f268413a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): get_auth_request con 0x7f268409fb50 auth_method 0 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): get_auth_request con 0x7f26840a0220 auth_method 0 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:41.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2683fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2688ff8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient(hunting): handle_auth_done global_id 4884 payload 824 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 -- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f268409fb50 msgr2=0x7f26841472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f268409fb50 0x7f26841472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 -- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0220 msgr2=0x7f268413a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0220 0x7f268413a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient: _finish_hunting 0 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 monclient: found mon.noname-d 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 -- 172.21.15.12:0/2735378640 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26700094f0 con 0x7f26840a0f80 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f26897f9700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f267c007f20 tx=0x7f267c009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f267c0098d0 con 0x7f26840a0f80 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:41.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: dump: 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:41.290 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: _finish_auth 0 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 5 monclient: authenticate success, global_id 4884 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c014420 con 0x7f26840a0f80 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f267c014a10 con 0x7f26840a0f80 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 10 monclient: _renew_subs 2023-09-29T23:18:41.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268404c960 con 0x7f26840a0f80 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2682ffd700 4 set_mon_vals no callback set 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f2680ff9700 10 monclient: dump: 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:41.292 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 10 monclient: _renew_subs 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:41.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.268+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2684126780 con 0x7f26840a0f80 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f267c014590 con 0x7f26840a0f80 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f268b25c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f268b25c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f268b25c700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2684070bc0 con 0x7f26840a0f80 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f2680ff9700 1 --2- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 0x7f2668038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.269+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f267c0477e0 con 0x7f26840a0f80 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.270+0000 7f2688ff8700 1 --2- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 0x7f2668038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.270+0000 7f2688ff8700 10 monclient: get_auth_request con 0x7f2668036850 auth_method 0 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.270+0000 7f2688ff8700 1 --2- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 0x7f2668038d00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2678006fd0 tx=0x7f2678008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.274+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f267c014840 con 0x7f26840a0f80 2023-09-29T23:18:41.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.274+0000 7f2680ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:41.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.274+0000 7f2680ff9700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:41.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f268b25c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:41.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f268b25c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:41.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f268b25c700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:41.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f268409ee90 con 0x7f26840a0f80 2023-09-29T23:18:41.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f2680ff9700 1 -- 172.21.15.12:0/2735378640 <== mon.6 v2:172.21.15.12:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f267c01b310 con 0x7f26840a0f80 2023-09-29T23:18:41.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f2680ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:41.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.510+0000 7f2680ff9700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:41.513 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:41.513 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:41.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 msgr2=0x7f2668038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:41.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 0x7f2668038d00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2678006fd0 tx=0x7f2678008040 comp rx=0 tx=0).stop 2023-09-29T23:18:41.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 10 monclient: shutdown 2023-09-29T23:18:41.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:41.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 msgr2=0x7f2684125480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f267c007f20 tx=0x7f267c009840 comp rx=0 tx=0).stop 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 shutdown_connections 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> v2:172.21.15.153:6812/83257 conn(0x7f2668036850 0x7f2668038d00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f26840a0f80 0x7f2684125480 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f268409fb50 0x7f26841472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 --2- 172.21.15.12:0/2735378640 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f26840a0220 0x7f268413a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 >> 172.21.15.12:0/2735378640 conn(0x7f26840a3080 msgr2=0x7f26840a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 shutdown_connections 2023-09-29T23:18:41.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:41.512+0000 7f268b25c700 1 -- 172.21.15.12:0/2735378640 wait complete. 2023-09-29T23:18:44.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 10 monclient: get_monmap_and_config 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 10 monclient: build_initial_monmap 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 10 monclient: monmap: 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:44.655975+0000 2023-09-29T23:18:44.655 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:44.655975+0000 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:44.656 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:44.657 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:44.657 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 1 Processor -- start 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 1 -- start start 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.655+0000 7f7fba589700 10 monclient: init 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient: _add_conns ranks=[9,1,10,4,5,6,3,2,0,8,7] 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7fb40a1f90 0x7f7fb409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-a con 0x7f7fb40a1f90 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7fb40a2360 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7fb4138980 0x7f7fb413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-f con 0x7f7fb4138980 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb4068ed0 con 0x7f7fb40a2360 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb409e350 con 0x7f7fb40a1f90 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb409ef70 con 0x7f7fb4138980 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fba589700 10 monclient(hunting): authenticate will time out at 1434.016380s 2023-09-29T23:18:44.666 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7fb40a1f90 0x7f7fb409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f7fb40a1f90 0x7f7fb409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51234/0 (socket says 172.21.15.12:51234) 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 -- 172.21.15.12:0/1274257370 learned_addr learned my addr 172.21.15.12:0/1274257370 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): get_auth_request con 0x7f7fb40a1f90 auth_method 0 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb8b26700 1 --2- 172.21.15.12:0/1274257370 >> v2:172.21.15.153:3300/0 conn(0x7f7fb4138980 0x7f7fb413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb8b26700 10 monclient(hunting): get_auth_request con 0x7f7fb4138980 auth_method 0 2023-09-29T23:18:44.667 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb8b26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb8b26700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb8b26700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_done global_id 5085 payload 274 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 -- 172.21.15.12:0/1274257370 >> v2:172.21.15.153:3300/0 conn(0x7f7fb4138980 msgr2=0x7f7fb413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 --2- 172.21.15.12:0/1274257370 >> v2:172.21.15.153:3300/0 conn(0x7f7fb4138980 0x7f7fb413ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 -- 172.21.15.12:0/1274257370 >> v1:172.21.15.153:6789/0 conn(0x7f7fb40a2360 legacy=0x7f7fb40a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 monclient: found mon.noname-a 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.656+0000 7f7fb3fff700 1 -- 172.21.15.12:0/1274257370 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb404c960 con 0x7f7fb40a1f90 2023-09-29T23:18:44.668 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb3fff700 1 --2- 172.21.15.12:0/1274257370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fb40a1f90 0x7f7fb409f5e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fb406b110 tx=0x7f7fa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428bb1183801c3cb server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 1 -- 172.21.15.12:0/1274257370 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fa400bce0 con 0x7f7fb40a1f90 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: dump: 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:44.669 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 5 monclient: authenticate success, global_id 5085 2023-09-29T23:18:44.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 1 -- 172.21.15.12:0/1274257370 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400be60 con 0x7f7fb40a1f90 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 4 set_mon_vals no callback set 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 1 -- 172.21.15.12:0/1274257370 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fa40118a0 con 0x7f7fb40a1f90 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fb2ffd700 10 monclient: dump: 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:44.671 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:44.672 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 10 monclient: shutdown 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 -- 172.21.15.12:0/1274257370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fb40a1f90 msgr2=0x7f7fb409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 --2- 172.21.15.12:0/1274257370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fb40a1f90 0x7f7fb409f5e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fb406b110 tx=0x7f7fa4009250 comp rx=0 tx=0).stop 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 -- 172.21.15.12:0/1274257370 shutdown_connections 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 --2- 172.21.15.12:0/1274257370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7fb40a1f90 0x7f7fb409f5e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 --2- 172.21.15.12:0/1274257370 >> v2:172.21.15.153:3300/0 conn(0x7f7fb4138980 0x7f7fb413ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 -- 172.21.15.12:0/1274257370 >> 172.21.15.12:0/1274257370 conn(0x7f7fb40a3b50 msgr2=0x7f7fb40a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 -- 172.21.15.12:0/1274257370 shutdown_connections 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.657+0000 7f7fba589700 1 -- 172.21.15.12:0/1274257370 wait complete. 2023-09-29T23:18:44.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.658+0000 7f7fba589700 10 monclient: build_initial_monmap 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.658+0000 7f7fba589700 10 monclient: monmap: 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:44.674 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.658+0000 7f7fba589700 1 Processor -- start 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.658+0000 7f7fba589700 1 -- start start 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient: init 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient: _add_conns ranks=[1,4,8,7,5,0,3,6,2] 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb40a1f90 0x7f7fb4097d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-f con 0x7f7fb40a1f90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-c con 0x7f7fb40a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 0x7f7fb408cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): picked mon.noname-e con 0x7f7fb4138980 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4090130 con 0x7f7fb40a1f90 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb409ffc0 con 0x7f7fb40a2360 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb409e500 con 0x7f7fb4138980 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:44.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fba589700 10 monclient(hunting): authenticate will time out at 1434.019446s 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 0x7f7fb408cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56580/0 (socket says 172.21.15.12:56580) 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 -- 172.21.15.12:0/181706624 learned_addr learned my addr 172.21.15.12:0/181706624 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb40a1f90 0x7f7fb4097d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): get_auth_request con 0x7f7fb40a2360 auth_method 0 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): get_auth_request con 0x7f7fb4138980 auth_method 0 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:44.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): get_auth_request con 0x7f7fb40a1f90 auth_method 0 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:44.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb8b26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient(hunting): handle_auth_done global_id 4927 payload 824 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 -- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 msgr2=0x7f7fb408cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 0x7f7fb408cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 -- 172.21.15.12:0/181706624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb40a1f90 msgr2=0x7f7fb4097d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb40a1f90 0x7f7fb4097d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 monclient: found mon.noname-c 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:44.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.659+0000 7f7fb37fe700 1 -- 172.21.15.12:0/181706624 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40094f0 con 0x7f7fb40a2360 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fb8b26700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 0x7f7fb408cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fb37fe700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c7c0 tx=0x7f7fa8007f40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fa8011a10 con 0x7f7fb40a2360 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: dump: 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.680 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:44.681 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: _finish_auth 0 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa8011b90 con 0x7f7fb40a2360 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 5 monclient: authenticate success, global_id 4927 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 10 monclient: _renew_subs 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb408fd50 con 0x7f7fb40a2360 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fa8011d00 con 0x7f7fb40a2360 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:44.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7f9bfff700 10 monclient: dump: 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:44.683 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 10 monclient: _renew_subs 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fb27fc700 4 set_mon_vals no callback set 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.660+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb408e990 con 0x7f7fb40a2360 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7fa8011590 con 0x7f7fb40a2360 2023-09-29T23:18:44.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fba589700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fba589700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fba589700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb4070bc0 con 0x7f7fb40a2360 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7f9bfff700 1 --2- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 0x7f7fa0038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7fa8046240 con 0x7f7fb40a2360 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fb3fff700 1 --2- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 0x7f7fa0038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.661+0000 7f7fb3fff700 10 monclient: get_auth_request con 0x7f7fa0036850 auth_method 0 2023-09-29T23:18:44.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.662+0000 7f7fb3fff700 1 --2- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 0x7f7fa0038d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400bfd0 tx=0x7f7fa4017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.666+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7fa8011840 con 0x7f7fb40a2360 2023-09-29T23:18:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.666+0000 7f7f9bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:44.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.666+0000 7f7f9bfff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.902+0000 7f7fba589700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.902+0000 7f7fba589700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.902+0000 7f7fba589700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.902+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7fb4090270 con 0x7f7fb40a2360 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.903+0000 7f7f9bfff700 1 -- 172.21.15.12:0/181706624 <== mon.4 v2:172.21.15.12:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7fa8017030 con 0x7f7fb40a2360 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.903+0000 7f7f9bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.903+0000 7f7f9bfff700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:44.903 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:18:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 msgr2=0x7f7fa0038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 0x7f7fa0038d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400bfd0 tx=0x7f7fa4017000 comp rx=0 tx=0).stop 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 10 monclient: shutdown 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 msgr2=0x7f7fb408c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c7c0 tx=0x7f7fa8007f40 comp rx=0 tx=0).stop 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 shutdown_connections 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> v2:172.21.15.153:6812/83257 conn(0x7f7fa0036850 0x7f7fa0038d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f7fb4138980 0x7f7fb408cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fb40a1f90 0x7f7fb4097d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 --2- 172.21.15.12:0/181706624 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f7fb40a2360 0x7f7fb408c680 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 >> 172.21.15.12:0/181706624 conn(0x7f7fb40a3b50 msgr2=0x7f7fb406b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 shutdown_connections 2023-09-29T23:18:44.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:44.905+0000 7f7fba589700 1 -- 172.21.15.12:0/181706624 wait complete. 2023-09-29T23:18:44.917 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-29T23:18:44.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 10 monclient: get_monmap_and_config 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 10 monclient: monmap: 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:45.035895+0000 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:45.035895+0000 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.035 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:45.036 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:45.037 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 1 Processor -- start 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 1 -- start start 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.035+0000 7f0f7e248700 10 monclient: init 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient: _add_conns ranks=[7,4,0,9,5,1,10,8,6,3,2] 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 0x7f0f78131740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-i con 0x7f0f78130fe0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 0x7f0f7806f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-c con 0x7f0f7806f0b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:45.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f780969e0 addr v1:172.21.15.12:6789/0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f78068ed0 con 0x7f0f780969e0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0f781313b0 con 0x7f0f7806f0b0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78131e00 con 0x7f0f78130fe0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f7e248700 10 monclient(hunting): authenticate will time out at 1434.396285s 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 0x7f0f7806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f77fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0f780969e0 0x7f0f78096e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33410/0 (socket says 172.21.15.12:33410) 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request con 0x7f0f7806f0b0 auth_method 0 2023-09-29T23:18:45.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f77fff700 1 -- 172.21.15.12:0/2261552570 learned_addr learned my addr 172.21.15.12:0/2261552570 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f777fe700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 0x7f0f78131740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_done global_id 4936 payload 274 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 -- 172.21.15.12:0/2261552570 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 msgr2=0x7f0f78131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 0x7f0f78131740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 -- 172.21.15.12:0/2261552570 >> v1:172.21.15.12:6789/0 conn(0x7f0f780969e0 legacy=0x7f0f78096e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f767fc700 1 -- 172.21.15.12:0/2261552570 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915617374 0 0) 0x7f0f78068ed0 con 0x7f0f780969e0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 monclient: found mon.noname-c 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 -- 172.21.15.12:0/2261552570 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7804c960 con 0x7f0f7806f0b0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f777fe700 10 monclient: get_auth_request con 0x7f0f78130fe0 auth_method 0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f777fe700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 0x7f0f78131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f767fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.036+0000 7f0f76ffd700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 0x7f0f7806f500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009910 tx=0x7f0f6c0091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bd23c7e47ba484c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 1 -- 172.21.15.12:0/2261552570 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0f6c00bce0 con 0x7f0f7806f0b0 2023-09-29T23:18:45.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: dump: 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.050 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 1 -- 172.21.15.12:0/2261552570 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c00be60 con 0x7f0f7806f0b0 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 5 monclient: authenticate success, global_id 4936 2023-09-29T23:18:45.051 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 1 -- 172.21.15.12:0/2261552570 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0f6c0118a0 con 0x7f0f7806f0b0 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 4 set_mon_vals no callback set 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f767fc700 10 monclient: dump: 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 10 monclient: shutdown 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 -- 172.21.15.12:0/2261552570 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 msgr2=0x7f0f7806f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 0x7f0f7806f500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009910 tx=0x7f0f6c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 -- 172.21.15.12:0/2261552570 shutdown_connections 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f78130fe0 0x7f0f78131740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 --2- 172.21.15.12:0/2261552570 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f0f7806f0b0 0x7f0f7806f500 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 -- 172.21.15.12:0/2261552570 >> 172.21.15.12:0/2261552570 conn(0x7f0f780a4430 msgr2=0x7f0f780a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 -- 172.21.15.12:0/2261552570 shutdown_connections 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.037+0000 7f0f7e248700 1 -- 172.21.15.12:0/2261552570 wait complete. 2023-09-29T23:18:45.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.038+0000 7f0f7e248700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.038+0000 7f0f7e248700 10 monclient: monmap: 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.055 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.038+0000 7f0f7e248700 1 Processor -- start 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.038+0000 7f0f7e248700 1 -- start start 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.038+0000 7f0f7e248700 10 monclient: init 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient: _add_conns ranks=[7,0,5,6,3,2,1,4,8] 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f7806f0b0 0x7f0f781422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-i con 0x7f0f7806f0b0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-a con 0x7f0f780969e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0f78130fe0 0x7f0f7806a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): picked mon.noname-h con 0x7f0f78130fe0 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780a05c0 con 0x7f0f780969e0 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78121710 con 0x7f0f78130fe0 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780a0180 con 0x7f0f7806f0b0 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f7e248700 10 monclient(hunting): authenticate will time out at 1434.399272s 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51242/0 (socket says 172.21.15.12:51242) 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 -- 172.21.15.12:0/3875445183 learned_addr learned my addr 172.21.15.12:0/3875445183 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request con 0x7f0f780969e0 auth_method 0 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0f78130fe0 0x7f0f7806a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f7806f0b0 0x7f0f781422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): get_auth_request con 0x7f0f78130fe0 auth_method 0 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): get_auth_request con 0x7f0f7806f0b0 auth_method 0 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient(hunting): handle_auth_done global_id 5103 payload 824 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 -- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f7806f0b0 msgr2=0x7f0f781422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f7806f0b0 0x7f0f781422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 -- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0f78130fe0 msgr2=0x7f0f7806a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0f78130fe0 0x7f0f7806a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 monclient: found mon.noname-a 2023-09-29T23:18:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 -- 172.21.15.12:0/3875445183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c009420 con 0x7f0f780969e0 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.039+0000 7f0f76ffd700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0098e0 tx=0x7f0f6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0f6c0118a0 con 0x7f0f780969e0 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: dump: 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.062 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c011a40 con 0x7f0f780969e0 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 5 monclient: authenticate success, global_id 5103 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 10 monclient: _renew_subs 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f78121430 con 0x7f0f780969e0 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0f6c011bb0 con 0x7f0f780969e0 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f75ffb700 4 set_mon_vals no callback set 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f5f7fe700 10 monclient: dump: 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.064 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 10 monclient: _renew_subs 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7806c5b0 con 0x7f0f780969e0 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.040+0000 7f0f7e248700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f7e248700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f7e248700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f78070bc0 con 0x7f0f780969e0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0f6c011d10 con 0x7f0f780969e0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f5f7fe700 1 --2- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 0x7f0f64038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0f6c048720 con 0x7f0f780969e0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.041+0000 7f0f777fe700 1 --2- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 0x7f0f64038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.042+0000 7f0f777fe700 10 monclient: get_auth_request con 0x7f0f64036850 auth_method 0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.042+0000 7f0f777fe700 1 --2- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 0x7f0f64038d00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800adc0 tx=0x7f0f68006f90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.046+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f6c02b710 con 0x7f0f780969e0 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.046+0000 7f0f5f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.046+0000 7f0f5f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:45.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.242+0000 7f0f7e248700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.242+0000 7f0f7e248700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.242+0000 7f0f7e248700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.242+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f78121120 con 0x7f0f780969e0 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.243+0000 7f0f5f7fe700 1 -- 172.21.15.12:0/3875445183 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+9978 (secure 0 0 0) 0x7f0f6c032090 con 0x7f0f780969e0 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.243+0000 7f0f5f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.243+0000 7f0f5f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:45.243 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:39.119641+0000","last_up_change":"2023-09-29T23:13:31.402198+0000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:13:32.778797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:13:36.755733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":83761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":83761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":83761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":83761}]},"public_addr":"172.21.15.12:6802/83761","cluster_addr":"172.21.15.12:6804/83761","heartbeat_back_addr":"172.21.15.12:6807/83761","heartbeat_front_addr":"172.21.15.12:6806/83761","state":["exists","up"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":83762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6801","nonce":83762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6805","nonce":83762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":83762}]},"public_addr":"172.21.15.12:6800/83762","cluster_addr":"172.21.15.12:6801/83762","heartbeat_back_addr":"172.21.15.12:6805/83762","heartbeat_front_addr":"172.21.15.12:6803/83762","state":["exists","up"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":83763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":83763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":83763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":83763}]},"public_addr":"172.21.15.12:6808/83763","cluster_addr":"172.21.15.12:6809/83763","heartbeat_back_addr":"172.21.15.12:6811/83763","heartbeat_front_addr":"172.21.15.12:6810/83763","state":["exists","up"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":83793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":83793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":83793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":83793}]},"public_addr":"172.21.15.153:6800/83793","cluster_addr":"172.21.15.153:6801/83793","heartbeat_back_addr":"172.21.15.153:6804/83793","heartbeat_front_addr":"172.21.15.153:6802/83793","state":["exists","up"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":83794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":83794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":83794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":83794}]},"public_addr":"172.21.15.153:6803/83794","cluster_addr":"172.21.15.153:6805/83794","heartbeat_back_addr":"172.21.15.153:6807/83794","heartbeat_front_addr":"172.21.15.153:6806/83794","state":["exists","up"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":83795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":83795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":83795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":83795}]},"public_addr":"172.21.15.153:6808/83795","cluster_addr":"172.21.15.153:6809/83795","heartbeat_back_addr":"172.21.15.153:6811/83795","heartbeat_front_addr":"172.21.15.153:6810/83795","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.551171+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.484006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.491551+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.520272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.501043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.509993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:18:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 msgr2=0x7f0f64038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 0x7f0f64038d00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800adc0 tx=0x7f0f68006f90 comp rx=0 tx=0).stop 2023-09-29T23:18:45.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 10 monclient: shutdown 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 msgr2=0x7f0f78142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0098e0 tx=0x7f0f6c017000 comp rx=0 tx=0).stop 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 shutdown_connections 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> v2:172.21.15.153:6812/83257 conn(0x7f0f64036850 0x7f0f64038d00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f0f78130fe0 0x7f0f7806a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f0f7806f0b0 0x7f0f781422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 --2- 172.21.15.12:0/3875445183 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f780969e0 0x7f0f78142800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 >> 172.21.15.12:0/3875445183 conn(0x7f0f780a4430 msgr2=0x7f0f7809d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 shutdown_connections 2023-09-29T23:18:45.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.245+0000 7f0f7e248700 1 -- 172.21.15.12:0/3875445183 wait complete. 2023-09-29T23:18:45.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-29T23:18:45.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:18:45.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.376+0000 7f48d10c8700 10 monclient: get_monmap_and_config 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.376+0000 7f48d10c8700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.376+0000 7f48d10c8700 10 monclient: monmap: 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:45.377345+0000 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:45.377345+0000 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.377 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:45.378 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 Processor -- start 2023-09-29T23:18:45.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 -- start start 2023-09-29T23:18:45.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient: init 2023-09-29T23:18:45.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient: _add_conns ranks=[9,4,10,8,7,1,3,2,5,0,6] 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48cc0698a0 0x7f48cc09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-a con 0x7f48cc0698a0 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-c con 0x7f48cc0a10c0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f48cc0a1a10 0x7f48cc139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-f con 0x7f48cc0a1a10 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc09d190 con 0x7f48cc0698a0 2023-09-29T23:18:45.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc0a23d0 con 0x7f48cc0a1a10 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc09ac90 con 0x7f48cc0a10c0 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48d10c8700 10 monclient(hunting): authenticate will time out at 1434.737583s 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56592/0 (socket says 172.21.15.12:56592) 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 1 -- 172.21.15.12:0/857439093 learned_addr learned my addr 172.21.15.12:0/857439093 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f48cc0698a0 0x7f48cc09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): get_auth_request con 0x7f48cc0a10c0 auth_method 0 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): get_auth_request con 0x7f48cc0698a0 auth_method 0 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cb59e700 1 --2- 172.21.15.12:0/857439093 >> v2:172.21.15.153:3300/0 conn(0x7f48cc0a1a10 0x7f48cc139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cb59e700 10 monclient(hunting): get_auth_request con 0x7f48cc0a1a10 auth_method 0 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cb59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cb59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient(hunting): handle_auth_done global_id 5112 payload 274 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 -- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 msgr2=0x7f48cc0a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:45.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 -- 172.21.15.12:0/857439093 >> v2:172.21.15.153:3300/0 conn(0x7f48cc0a1a10 msgr2=0x7f48cc139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 --2- 172.21.15.12:0/857439093 >> v2:172.21.15.153:3300/0 conn(0x7f48cc0a1a10 0x7f48cc139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 monclient: found mon.noname-a 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48cad9d700 1 -- 172.21.15.12:0/857439093 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc068ed0 con 0x7f48cc0698a0 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.377+0000 7f48ca59c700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48cad9d700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48cc0698a0 0x7f48cc09cc50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0098e0 tx=0x7f48bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2ce1f3709bbb26 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 1 -- 172.21.15.12:0/857439093 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48bc011ab0 con 0x7f48cc0698a0 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: dump: 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.393 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 1 -- 172.21.15.12:0/857439093 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc011c30 con 0x7f48cc0698a0 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 5 monclient: authenticate success, global_id 5112 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 1 -- 172.21.15.12:0/857439093 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48bc011da0 con 0x7f48cc0698a0 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 4 set_mon_vals no callback set 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48c9d9b700 10 monclient: dump: 2023-09-29T23:18:45.395 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.396 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 10 monclient: shutdown 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 -- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48cc0698a0 msgr2=0x7f48cc09cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48cc0698a0 0x7f48cc09cc50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0098e0 tx=0x7f48bc0091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 -- 172.21.15.12:0/857439093 shutdown_connections 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 --2- 172.21.15.12:0/857439093 >> v2:172.21.15.153:3300/0 conn(0x7f48cc0a1a10 0x7f48cc139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48cc0698a0 0x7f48cc09cc50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 --2- 172.21.15.12:0/857439093 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f48cc0a10c0 0x7f48cc0a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 -- 172.21.15.12:0/857439093 >> 172.21.15.12:0/857439093 conn(0x7f48cc063d70 msgr2=0x7f48cc09fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 -- 172.21.15.12:0/857439093 shutdown_connections 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.378+0000 7f48d10c8700 1 -- 172.21.15.12:0/857439093 wait complete. 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.379+0000 7f48d10c8700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.379+0000 7f48d10c8700 10 monclient: monmap: 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:45.398 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.399 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.379+0000 7f48d10c8700 1 Processor -- start 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.379+0000 7f48d10c8700 1 -- start start 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.379+0000 7f48d10c8700 10 monclient: init 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient: _add_conns ranks=[7,1,5,6,3,4,2,8,0] 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-i con 0x7f48cc0698a0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-f con 0x7f48cc0a10c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): picked mon.noname-h con 0x7f48cc0a1a10 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc1203b0 con 0x7f48cc0a10c0 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc06ecf0 con 0x7f48cc0a1a10 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc143290 con 0x7f48cc0698a0 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48d10c8700 10 monclient(hunting): authenticate will time out at 1434.740386s 2023-09-29T23:18:45.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:55364/0 (socket says 172.21.15.12:55364) 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.12:51998/0 (socket says 172.21.15.12:51998) 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 -- 172.21.15.12:0/3377624214 learned_addr learned my addr 172.21.15.12:0/3377624214 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:45492/0 (socket says 172.21.15.12:45492) 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): get_auth_request con 0x7f48cc0a1a10 auth_method 0 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): get_auth_request con 0x7f48cc0698a0 auth_method 0 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): get_auth_request con 0x7f48cc0a10c0 auth_method 0 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 -- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 msgr2=0x7f48cc1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 -- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 msgr2=0x7f48cc132fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 monclient: found mon.noname-i 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.153:3303/0 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.380+0000 7f48cad9d700 1 -- 172.21.15.12:0/3377624214 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c00094f0 con 0x7f48cc0698a0 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48ca59c700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48cb59e700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48cad9d700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48cc09d2d0 tx=0x7f48bc017000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48bc01f070 con 0x7f48cc0698a0 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-29T23:18:45.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: dump: 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.406 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 5 monclient: authenticate success, global_id 4732 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc01add0 con 0x7f48cc0698a0 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 10 monclient: _renew_subs 2023-09-29T23:18:45.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc009420 con 0x7f48cc0698a0 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48bc0268f0 con 0x7f48cc0698a0 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48c959a700 4 set_mon_vals no callback set 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48b2ffd700 10 monclient: dump: 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.408 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.409 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 10 monclient: _renew_subs 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.381+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc04c960 con 0x7f48cc0698a0 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f48bc023070 con 0x7f48cc0698a0 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48d10c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48d10c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48d10c8700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.153:3303/0 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc070bc0 con 0x7f48cc0698a0 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48b2ffd700 1 --2- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 0x7f48b8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f48bc048520 con 0x7f48cc0698a0 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48ca59c700 1 --2- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 0x7f48b8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.382+0000 7f48ca59c700 10 monclient: get_auth_request con 0x7f48b8036850 auth_method 0 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.383+0000 7f48ca59c700 1 --2- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 0x7f48b8038d00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006d40 tx=0x7f48c0009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.387+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mon.7 v2:172.21.15.153:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48bc024370 con 0x7f48cc0698a0 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.387+0000 7f48b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.387+0000 7f48b2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:45.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.577+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48cc069190 con 0x7f48b8036850 2023-09-29T23:18:45.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.580+0000 7f48b2ffd700 1 -- 172.21.15.12:0/3377624214 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52217 (secure 0 0 0) 0x7f48c0011eb0 con 0x7f48b8036850 2023-09-29T23:18:45.581 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:45.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 msgr2=0x7f48b8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 0x7f48b8038d00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006d40 tx=0x7f48c0009250 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 10 monclient: shutdown 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 msgr2=0x7f48cc06df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48cc09d2d0 tx=0x7f48bc017000 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 shutdown_connections 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> v2:172.21.15.153:6812/83257 conn(0x7f48b8036850 0x7f48b8038d00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f48cc0a1a10 0x7f48cc1334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f48cc0698a0 0x7f48cc06df90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 --2- 172.21.15.12:0/3377624214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc0a10c0 0x7f48cc132fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 >> 172.21.15.12:0/3377624214 conn(0x7f48cc063d70 msgr2=0x7f48cc09fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 shutdown_connections 2023-09-29T23:18:45.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.582+0000 7f48d10c8700 1 -- 172.21.15.12:0/3377624214 wait complete. 2023-09-29T23:18:45.584 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:18:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-29T23:18:44.791246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4777096,"kb_used_data":557419184,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557419184,"statfs":{"total":575688990720,"available":570797244416,"internally_reserved":0,"allocated":570797244416,"data_stored":570797244416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422822,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.129207+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:38.129207+0000","last_peered":"2023-09-29T23:13:38.129207+0000","last_clean":"2023-09-29T23:13:38.129207+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:38.129207+0000","last_undegraded":"2023-09-29T23:13:38.129207+0000","last_fullsized":"2023-09-29T23:13:38.129207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:35.802738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.130672+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:38.130672+0000","last_peered":"2023-09-29T23:13:38.130672+0000","last_clean":"2023-09-29T23:13:38.130672+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:38.130672+0000","last_undegraded":"2023-09-29T23:13:38.130672+0000","last_fullsized":"2023-09-29T23:13:38.130672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:01.042284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127425+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:38.127425+0000","last_peered":"2023-09-29T23:13:38.127425+0000","last_clean":"2023-09-29T23:13:38.127425+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:38.127425+0000","last_undegraded":"2023-09-29T23:13:38.127425+0000","last_fullsized":"2023-09-29T23:13:38.127425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:56.316625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.128695+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:38.128695+0000","last_peered":"2023-09-29T23:13:38.128695+0000","last_clean":"2023-09-29T23:13:38.128695+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:38.128695+0000","last_undegraded":"2023-09-29T23:13:38.128695+0000","last_fullsized":"2023-09-29T23:13:38.128695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:59:39.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126113+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:38.126113+0000","last_peered":"2023-09-29T23:13:38.126113+0000","last_clean":"2023-09-29T23:13:38.126113+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:38.126113+0000","last_undegraded":"2023-09-29T23:13:38.126113+0000","last_fullsized":"2023-09-29T23:13:38.126113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:24.806629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119092,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.41299999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.62,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.504}]},{"osd":1,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.41599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63800000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65000000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.66200000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45700000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.51400000000000001,"5min":0.627,"15min":0.627},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.502,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.439}]}]},{"osd":4,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.50800000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60799999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.44},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.33200000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.373},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.27800000000000002}]},{"osd":5,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59599999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48299999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118413,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34200000000000003,"15min":0.34399999999999997},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.376,"15min":0.379},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.41299999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23400000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48499999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.254,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.44800000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.432}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.41499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.622}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.432,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70399999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.71399999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68799999999999994,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65400000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43684,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.36399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.62,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.316}]},{"osd":1,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69699999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.40600000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67200000000000004,"5min":0.754,"15min":0.754},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":0.64400000000000002},"min":{"1min":0.432,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71499999999999997,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.436}]},{"osd":4,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57799999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58299999999999996,"5min":0.63,"15min":0.63},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.46700000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46200000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.48699999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.35499999999999998},"min":{"1min":0.26300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.44400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.316,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78400000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.624}]},{"osd":4,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.44600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.47899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79100000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57299999999999995,"15min":0.60999999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43703,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.437,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.308},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.35599999999999998,"15min":0.34300000000000003},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59099999999999997,"5min":1.115,"15min":1.115},"last":0.245}]},{"osd":1,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44900000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.36299999999999999,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46999999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73299999999999998,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75900000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69099999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79700000000000004,"5min":1.601,"15min":1.601},"last":0.626}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77200000000000002,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74199999999999999,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.71499999999999997}]}]}],"pool_statfs":[]}} 2023-09-29T23:18:45.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:18:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.713+0000 7f383a680700 10 monclient: get_monmap_and_config 2023-09-29T23:18:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.713+0000 7f383a680700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.713+0000 7f383a680700 10 monclient: monmap: 2023-09-29T23:18:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.713 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:45.714164+0000 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:45.714164+0000 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.714 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:45.715 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:45.715 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 Processor -- start 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 -- start start 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient: init 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient: _add_conns ranks=[1,5,3,6,7,0,9,10,4,8,2] 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f38340a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f38340a2360 0x7f38340a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): picked mon.noname-h con 0x7f38340a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 0x7f383413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): picked mon.noname-g con 0x7f3834138980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3834068ed0 con 0x7f38340a1f90 2023-09-29T23:18:45.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383409e350 con 0x7f3834138980 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f383409fcf0 con 0x7f38340a2360 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f383a680700 10 monclient(hunting): authenticate will time out at 1435.074522s 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3833fff700 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f38340a1f90 0x7f383409f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:37550/0 (socket says 172.21.15.12:37550) 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 0x7f383413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3833fff700 1 -- 172.21.15.12:0/2147039607 learned_addr learned my addr 172.21.15.12:0/2147039607 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): get_auth_request con 0x7f3834138980 auth_method 0 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.724 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f38340a2360 0x7f38340a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): get_auth_request con 0x7f38340a2360 auth_method 0 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.714+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 10 monclient(hunting): handle_auth_done global_id 4728 payload 274 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 -- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f38340a2360 msgr2=0x7f38340a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f38340a2360 0x7f38340a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 1 -- 172.21.15.12:0/2147039607 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576921833 0 0) 0x7f3834068ed0 con 0x7f38340a1f90 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 -- 172.21.15.12:0/2147039607 >> v1:172.21.15.153:6789/0 conn(0x7f38340a1f90 legacy=0x7f383409f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 monclient: found mon.noname-g 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.153:3301/0 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 -- 172.21.15.12:0/2147039607 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383404c960 con 0x7f3834138980 2023-09-29T23:18:45.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:18:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3838c1d700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 0x7f383413ad60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f382c009e90 tx=0x7f382c009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=53673e7df0d3e6a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 1 -- 172.21.15.12:0/2147039607 <== mon.3 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f382c00e070 con 0x7f3834138980 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: dump: 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.729 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 5 monclient: authenticate success, global_id 4728 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 1 -- 172.21.15.12:0/2147039607 <== mon.3 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00bbd0 con 0x7f3834138980 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 1 -- 172.21.15.12:0/2147039607 <== mon.3 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f382c00bd40 con 0x7f3834138980 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 4 set_mon_vals no callback set 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f3832ffd700 10 monclient: dump: 2023-09-29T23:18:45.731 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.732 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 10 monclient: shutdown 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 1 -- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 msgr2=0x7f383413ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.715+0000 7f383a680700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 0x7f383413ad60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f382c009e90 tx=0x7f382c009780 comp rx=0 tx=0).stop 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 -- 172.21.15.12:0/2147039607 shutdown_connections 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3834138980 0x7f383413ad60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 --2- 172.21.15.12:0/2147039607 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f38340a2360 0x7f38340a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 -- 172.21.15.12:0/2147039607 >> 172.21.15.12:0/2147039607 conn(0x7f38340a3b50 msgr2=0x7f38340a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 -- 172.21.15.12:0/2147039607 shutdown_connections 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 -- 172.21.15.12:0/2147039607 wait complete. 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 10 monclient: build_initial_monmap 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 10 monclient: monmap: 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:45.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.716+0000 7f383a680700 1 Processor -- start 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 -- start start 2023-09-29T23:18:45.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient: init 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient: _add_conns ranks=[3,1,0,7,2,5,4,6,8] 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38340a1f90 0x7f3834142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): picked mon.noname-g con 0x7f38340a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 0x7f3834097810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): picked mon.noname-f con 0x7f38340a2360 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): picked mon.noname-a con 0x7f3834138980 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340a0030 con 0x7f3834138980 2023-09-29T23:18:45.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383408ffa0 con 0x7f38340a2360 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383408fcc0 con 0x7f38340a1f90 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f383a680700 10 monclient(hunting): authenticate will time out at 1435.077543s 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51252/0 (socket says 172.21.15.12:51252) 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 1 -- 172.21.15.12:0/3701177893 learned_addr learned my addr 172.21.15.12:0/3701177893 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): get_auth_request con 0x7f3834138980 auth_method 0 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38340a1f90 0x7f3834142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 0x7f3834097810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): get_auth_request con 0x7f38340a2360 auth_method 0 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): get_auth_request con 0x7f38340a1f90 auth_method 0 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f38337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3833fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.717+0000 7f3838c1d700 10 monclient(hunting): handle_auth_done global_id 5121 payload 824 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 -- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38340a1f90 msgr2=0x7f3834142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38340a1f90 0x7f3834142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 -- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 msgr2=0x7f3834097810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 0x7f3834097810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 monclient: found mon.noname-a 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 -- 172.21.15.12:0/3701177893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c009a20 con 0x7f3834138980 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f38337fe700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 0x7f3834097810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f3838c1d700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b6c0 tx=0x7f382c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f382c0070c0 con 0x7f3834138980 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:45.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: dump: 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.741 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: _finish_auth 0 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c0263f0 con 0x7f3834138980 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 5 monclient: authenticate success, global_id 5121 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 10 monclient: _renew_subs 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383408f9e0 con 0x7f3834138980 2023-09-29T23:18:45.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f382c0269e0 con 0x7f3834138980 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f381bfff700 10 monclient: dump: 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:45.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f38327fc700 4 set_mon_vals no callback set 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 10 monclient: _renew_subs 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.718+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3834098290 con 0x7f3834138980 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.719+0000 7f383a680700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.719+0000 7f383a680700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.719+0000 7f383a680700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.719+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834070bc0 con 0x7f3834138980 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f382c026560 con 0x7f3834138980 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f381bfff700 1 --2- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 0x7f3820038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f382c0493c0 con 0x7f3834138980 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f3833fff700 1 --2- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 0x7f3820038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f3833fff700 10 monclient: get_auth_request con 0x7f3820036850 auth_method 0 2023-09-29T23:18:45.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.720+0000 7f3833fff700 1 --2- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 0x7f3820038d00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3828007fd0 tx=0x7f3828007d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.724+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f382c02e360 con 0x7f3834138980 2023-09-29T23:18:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.724+0000 7f381bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.724+0000 7f381bfff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:45.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.910+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3834069190 con 0x7f3820036850 2023-09-29T23:18:45.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.912+0000 7f381bfff700 1 -- 172.21.15.12:0/3701177893 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52217 (secure 0 0 0) 0x7f3828001400 con 0x7f3820036850 2023-09-29T23:18:45.913 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 msgr2=0x7f3820038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 0x7f3820038d00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3828007fd0 tx=0x7f3828007d10 comp rx=0 tx=0).stop 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 10 monclient: shutdown 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 msgr2=0x7f3834097d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.914+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b6c0 tx=0x7f382c019000 comp rx=0 tx=0).stop 2023-09-29T23:18:45.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 shutdown_connections 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3834138980 0x7f3834097d50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> v2:172.21.15.153:6812/83257 conn(0x7f3820036850 0x7f3820038d00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38340a1f90 0x7f3834142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 --2- 172.21.15.12:0/3701177893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38340a2360 0x7f3834097810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 >> 172.21.15.12:0/3701177893 conn(0x7f38340a3b50 msgr2=0x7f38340a5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 shutdown_connections 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:45.915+0000 7f383a680700 1 -- 172.21.15.12:0/3701177893 wait complete. 2023-09-29T23:18:45.916 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:18:45.926 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-29T23:18:44.791246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4777096,"kb_used_data":557419184,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557419184,"statfs":{"total":575688990720,"available":570797244416,"internally_reserved":0,"allocated":570797244416,"data_stored":570797244416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422822,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.129207+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:38.129207+0000","last_peered":"2023-09-29T23:13:38.129207+0000","last_clean":"2023-09-29T23:13:38.129207+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:38.129207+0000","last_undegraded":"2023-09-29T23:13:38.129207+0000","last_fullsized":"2023-09-29T23:13:38.129207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:35.802738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.130672+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:38.130672+0000","last_peered":"2023-09-29T23:13:38.130672+0000","last_clean":"2023-09-29T23:13:38.130672+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:38.130672+0000","last_undegraded":"2023-09-29T23:13:38.130672+0000","last_fullsized":"2023-09-29T23:13:38.130672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:01.042284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127425+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:38.127425+0000","last_peered":"2023-09-29T23:13:38.127425+0000","last_clean":"2023-09-29T23:13:38.127425+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:38.127425+0000","last_undegraded":"2023-09-29T23:13:38.127425+0000","last_fullsized":"2023-09-29T23:13:38.127425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:56.316625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.128695+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:38.128695+0000","last_peered":"2023-09-29T23:13:38.128695+0000","last_clean":"2023-09-29T23:13:38.128695+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:38.128695+0000","last_undegraded":"2023-09-29T23:13:38.128695+0000","last_fullsized":"2023-09-29T23:13:38.128695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:59:39.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126113+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:38.126113+0000","last_peered":"2023-09-29T23:13:38.126113+0000","last_clean":"2023-09-29T23:13:38.126113+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:38.126113+0000","last_undegraded":"2023-09-29T23:13:38.126113+0000","last_fullsized":"2023-09-29T23:13:38.126113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:24.806629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119092,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.41299999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.62,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.504}]},{"osd":1,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.41599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63800000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65000000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.66200000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45700000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.51400000000000001,"5min":0.627,"15min":0.627},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.502,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.439}]}]},{"osd":4,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.50800000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60799999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.44},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.33200000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.373},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.27800000000000002}]},{"osd":5,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59599999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48299999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118413,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34200000000000003,"15min":0.34399999999999997},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.376,"15min":0.379},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.41299999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23400000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48499999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.254,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.44800000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.432}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.41499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.622}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.432,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70399999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.71399999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68799999999999994,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65400000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43684,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.36399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.62,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.316}]},{"osd":1,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69699999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.40600000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67200000000000004,"5min":0.754,"15min":0.754},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":0.64400000000000002},"min":{"1min":0.432,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71499999999999997,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.436}]},{"osd":4,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57799999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58299999999999996,"5min":0.63,"15min":0.63},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.46700000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46200000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.48699999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.35499999999999998},"min":{"1min":0.26300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.44400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.316,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78400000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.624}]},{"osd":4,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.44600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.47899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79100000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57299999999999995,"15min":0.60999999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43703,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.437,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.308},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.35599999999999998,"15min":0.34300000000000003},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59099999999999997,"5min":1.115,"15min":1.115},"last":0.245}]},{"osd":1,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44900000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.36299999999999999,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46999999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73299999999999998,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75900000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69099999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79700000000000004,"5min":1.601,"15min":1.601},"last":0.626}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77200000000000002,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74199999999999999,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.71499999999999997}]}]}],"pool_statfs":[]}} 2023-09-29T23:18:45.928 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-29T23:18:45.929 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:18:46.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.046+0000 7f178b6f8700 10 monclient: get_monmap_and_config 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.046+0000 7f178b6f8700 10 monclient: build_initial_monmap 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.046+0000 7f178b6f8700 10 monclient: monmap: 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:46.047153+0000 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:46.047153+0000 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:46.047 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.046+0000 7f178b6f8700 1 Processor -- start 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 -- start start 2023-09-29T23:18:46.048 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient: init 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient: _add_conns ranks=[8,5,6,9,4,7,1,3,2,0,10] 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-e con 0x7f17840a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f17840a2360 0x7f17840a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-h con 0x7f17840a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-d con 0x7f1784133740 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:46.056 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f178409e350 con 0x7f17840a2360 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f178409ef70 con 0x7f1784133740 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f178409da70 con 0x7f17840a1f90 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f178b6f8700 10 monclient(hunting): authenticate will time out at 1435.407538s 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55254/0 (socket says 172.21.15.12:55254) 2023-09-29T23:18:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37112/0 (socket says 172.21.15.12:37112) 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 1 -- 172.21.15.12:0/1659073926 learned_addr learned my addr 172.21.15.12:0/1659073926 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): get_auth_request con 0x7f17840a1f90 auth_method 0 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): get_auth_request con 0x7f1784133740 auth_method 0 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789c95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1788c93700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f17840a2360 0x7f17840a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1788c93700 10 monclient(hunting): get_auth_request con 0x7f17840a2360 auth_method 0 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1788c93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1788c93700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1788c93700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.047+0000 7f1789494700 10 monclient(hunting): handle_auth_done global_id 4787 payload 274 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 -- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 msgr2=0x7f1784133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.059 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 -- 172.21.15.12:0/1659073926 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f17840a2360 msgr2=0x7f17840a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f17840a2360 0x7f17840a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 10 monclient: _finish_hunting 0 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 monclient: found mon.noname-e 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 -- 172.21.15.12:0/1659073926 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784068ed0 con 0x7f17840a1f90 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789c95700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1789494700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fb0 tx=0x7f1774009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=fb0b70df6c165cb server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 1 -- 172.21.15.12:0/1659073926 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f177400e070 con 0x7f17840a1f90 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: dump: 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.061 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: _finish_auth 0 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 1 -- 172.21.15.12:0/1659073926 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177400be80 con 0x7f17840a1f90 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 5 monclient: authenticate success, global_id 4787 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:46.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 1 -- 172.21.15.12:0/1659073926 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f17740092e0 con 0x7f17840a1f90 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 4 set_mon_vals no callback set 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f1783fff700 10 monclient: dump: 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.063 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 10 monclient: shutdown 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 -- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 msgr2=0x7f178409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fb0 tx=0x7f1774009840 comp rx=0 tx=0).stop 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 -- 172.21.15.12:0/1659073926 shutdown_connections 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f1784133740 0x7f1784133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f17840a1f90 0x7f178409f5e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 --2- 172.21.15.12:0/1659073926 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f17840a2360 0x7f17840a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.048+0000 7f178b6f8700 1 -- 172.21.15.12:0/1659073926 >> 172.21.15.12:0/1659073926 conn(0x7f17840a3b50 msgr2=0x7f17840a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 1 -- 172.21.15.12:0/1659073926 shutdown_connections 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 1 -- 172.21.15.12:0/1659073926 wait complete. 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 10 monclient: build_initial_monmap 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 10 monclient: monmap: 2023-09-29T23:18:46.065 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:46.066 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 1 Processor -- start 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 1 -- start start 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.049+0000 7f178b6f8700 10 monclient: init 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient: _add_conns ranks=[0,3,5,7,4,2,1,6,8] 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-a con 0x7f17840a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17840a2360 0x7f1784142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-g con 0x7f17840a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1784133740 0x7f17841428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): picked mon.noname-h con 0x7f1784133740 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178409e490 con 0x7f17840a1f90 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1784131060 con 0x7f17840a2360 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1784130d80 con 0x7f1784133740 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f178b6f8700 10 monclient(hunting): authenticate will time out at 1435.410527s 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51260/0 (socket says 172.21.15.12:51260) 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 -- 172.21.15.12:0/3961152758 learned_addr learned my addr 172.21.15.12:0/3961152758 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): get_auth_request con 0x7f17840a1f90 auth_method 0 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient(hunting): handle_auth_done global_id 5130 payload 824 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1788c93700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17840a2360 0x7f1784142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789c95700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1784133740 0x7f17841428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 -- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1784133740 msgr2=0x7f17841428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1784133740 0x7f17841428a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 -- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17840a2360 msgr2=0x7f1784142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17840a2360 0x7f1784142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient: _finish_hunting 0 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 monclient: found mon.noname-a 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.050+0000 7f1789494700 1 -- 172.21.15.12:0/3961152758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c009420 con 0x7f17840a1f90 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f1789494700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1774009df0 tx=0x7f1774018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.070 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f17740092e0 con 0x7f17840a1f90 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: dump: 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.071 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1774007000 con 0x7f17840a1f90 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 5 monclient: authenticate success, global_id 5130 2023-09-29T23:18:46.072 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 10 monclient: _renew_subs 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1774009ae0 con 0x7f17840a1f90 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1774026d60 con 0x7f17840a1f90 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 10 monclient: dump: 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.073 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17837fe700 4 set_mon_vals no callback set 2023-09-29T23:18:46.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 10 monclient: _renew_subs 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178404c960 con 0x7f17840a1f90 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.051+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1774003040 con 0x7f17840a1f90 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f17817fa700 1 --2- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 0x7f1770038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1774048cb0 con 0x7f17840a1f90 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f178b6f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f178b6f8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f178b6f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1784070bc0 con 0x7f17840a1f90 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f1788c93700 1 --2- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 0x7f1770038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f1788c93700 10 monclient: get_auth_request con 0x7f1770036780 auth_method 0 2023-09-29T23:18:46.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.052+0000 7f1788c93700 1 --2- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 0x7f1770038c30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1778006fd0 tx=0x7f1778008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.056+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1774021020 con 0x7f17840a1f90 2023-09-29T23:18:46.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.056+0000 7f17817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.056+0000 7f17817fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:46.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.251+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1784069190 con 0x7f1770036780 2023-09-29T23:18:46.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.253+0000 7f17817fa700 1 -- 172.21.15.12:0/3961152758 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52217 (secure 0 0 0) 0x7f1778009e70 con 0x7f1770036780 2023-09-29T23:18:46.254 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 msgr2=0x7f1770038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 0x7f1770038c30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1778006fd0 tx=0x7f1778008040 comp rx=0 tx=0).stop 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 10 monclient: shutdown 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 msgr2=0x7f1784132b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1774009df0 tx=0x7f1774018000 comp rx=0 tx=0).stop 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 shutdown_connections 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f1784133740 0x7f17841428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17840a1f90 0x7f1784132b50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> v2:172.21.15.153:6812/83257 conn(0x7f1770036780 0x7f1770038c30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 --2- 172.21.15.12:0/3961152758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17840a2360 0x7f1784142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.255+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 >> 172.21.15.12:0/3961152758 conn(0x7f17840a3b50 msgr2=0x7f17840a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.256+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 shutdown_connections 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.256+0000 7f178b6f8700 1 -- 172.21.15.12:0/3961152758 wait complete. 2023-09-29T23:18:46.257 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:18:46.267 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-29T23:18:44.791246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4777096,"kb_used_data":557419184,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557419184,"statfs":{"total":575688990720,"available":570797244416,"internally_reserved":0,"allocated":570797244416,"data_stored":570797244416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422822,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.129207+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:38.129207+0000","last_peered":"2023-09-29T23:13:38.129207+0000","last_clean":"2023-09-29T23:13:38.129207+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:38.129207+0000","last_undegraded":"2023-09-29T23:13:38.129207+0000","last_fullsized":"2023-09-29T23:13:38.129207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:58:35.802738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.130672+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:38.130672+0000","last_peered":"2023-09-29T23:13:38.130672+0000","last_clean":"2023-09-29T23:13:38.130672+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:38.130672+0000","last_undegraded":"2023-09-29T23:13:38.130672+0000","last_fullsized":"2023-09-29T23:13:38.130672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:01.042284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127425+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:38.127425+0000","last_peered":"2023-09-29T23:13:38.127425+0000","last_clean":"2023-09-29T23:13:38.127425+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:38.127425+0000","last_undegraded":"2023-09-29T23:13:38.127425+0000","last_fullsized":"2023-09-29T23:13:38.127425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:56.316625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.128695+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:38.128695+0000","last_peered":"2023-09-29T23:13:38.128695+0000","last_clean":"2023-09-29T23:13:38.128695+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:38.128695+0000","last_undegraded":"2023-09-29T23:13:38.128695+0000","last_fullsized":"2023-09-29T23:13:38.128695+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:59:39.676061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126113+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:38.126113+0000","last_peered":"2023-09-29T23:13:38.126113+0000","last_clean":"2023-09-29T23:13:38.126113+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:38.126113+0000","last_undegraded":"2023-09-29T23:13:38.126113+0000","last_fullsized":"2023-09-29T23:13:38.126113+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:24.806629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119092,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.41299999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.62,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.504}]},{"osd":1,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.41599999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63800000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65000000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52000000000000002},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.66200000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45700000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.51400000000000001,"5min":0.627,"15min":0.627},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Sep 29 23:17:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.502,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.439}]}]},{"osd":4,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.50800000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60799999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.44},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.33200000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.373},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.27800000000000002}]},{"osd":5,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59599999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48299999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118413,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.34200000000000003,"15min":0.34399999999999997},"min":{"1min":0.22700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.376,"15min":0.379},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.502,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.41299999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23400000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48499999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.254,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.44800000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.432}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.43099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.41499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.622}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.432,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70399999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.71399999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68799999999999994,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65400000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43684,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.49199999999999999},"min":{"1min":0.36399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.62,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.316}]},{"osd":1,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69699999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.40600000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67200000000000004,"5min":0.754,"15min":0.754},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57699999999999996,"15min":0.64400000000000002},"min":{"1min":0.432,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71499999999999997,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.436}]},{"osd":4,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41199999999999998,"15min":0.41899999999999998},"min":{"1min":0.29799999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57799999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Sep 29 23:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58299999999999996,"5min":0.63,"15min":0.63},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.46700000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46200000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.48699999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.35499999999999998},"min":{"1min":0.26300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.44400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.316,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78400000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.624}]},{"osd":4,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.44600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.47899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79100000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57299999999999995,"15min":0.60999999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43703,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.33400000000000002,"15min":0.32800000000000001},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.437,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.308},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.35599999999999998,"15min":0.34300000000000003},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59099999999999997,"5min":1.115,"15min":1.115},"last":0.245}]},{"osd":1,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44900000000000001,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.36299999999999999,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46999999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73299999999999998,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75900000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69099999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.38100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79700000000000004,"5min":1.601,"15min":1.601},"last":0.626}]},{"osd":5,"last update":"Fri Sep 29 23:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77200000000000002,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74199999999999999,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.71499999999999997}]}]}],"pool_statfs":[]}} 2023-09-29T23:18:46.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.387+0000 7ff27f2df700 10 monclient: get_monmap_and_config 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.387+0000 7ff27f2df700 10 monclient: build_initial_monmap 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.388+0000 7ff27f2df700 10 monclient: monmap: 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:46.388850+0000 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:46.388850+0000 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.388 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:46.389 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.388+0000 7ff27f2df700 1 Processor -- start 2023-09-29T23:18:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.388+0000 7ff27f2df700 1 -- start start 2023-09-29T23:18:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.388+0000 7ff27f2df700 10 monclient: init 2023-09-29T23:18:46.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient: _add_conns ranks=[8,6,10,4,7,9,0,1,5,3,2] 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-e con 0x7ff2780a1f90 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-d con 0x7ff2780a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 0x7ff278133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-f con 0x7ff278133740 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27809e350 con 0x7ff278133740 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff27809ef70 con 0x7ff2780a2360 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff27809da70 con 0x7ff2780a1f90 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27f2df700 10 monclient(hunting): authenticate will time out at 1435.749223s 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37118/0 (socket says 172.21.15.12:37118) 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55256/0 (socket says 172.21.15.12:55256) 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 -- 172.21.15.12:0/3605041200 learned_addr learned my addr 172.21.15.12:0/3605041200 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): get_auth_request con 0x7ff2780a2360 auth_method 0 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): get_auth_request con 0x7ff2780a1f90 auth_method 0 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d87c700 1 --2- 172.21.15.12:0/3605041200 >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 0x7ff278133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient(hunting): handle_auth_done global_id 4902 payload 274 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 -- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 msgr2=0x7ff27809f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 -- 172.21.15.12:0/3605041200 >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 msgr2=0x7ff278133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 --2- 172.21.15.12:0/3605041200 >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 0x7ff278133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient: _finish_hunting 0 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 monclient: found mon.noname-d 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 -- 172.21.15.12:0/3605041200 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278068ed0 con 0x7ff2780a2360 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d87c700 10 monclient: get_auth_request con 0x7ff278133740 auth_method 0 2023-09-29T23:18:46.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d07b700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27d87c700 1 --2- 172.21.15.12:0/3605041200 >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 0x7ff278133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff27c87a700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2740099b0 tx=0x7ff274009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=858c1b3b4c716495 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.389+0000 7ff26ffff700 1 -- 172.21.15.12:0/3605041200 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff274009ea0 con 0x7ff2780a2360 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: dump: 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.402 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: _finish_auth 0 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 1 -- 172.21.15.12:0/3605041200 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27400bce0 con 0x7ff2780a2360 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 5 monclient: authenticate success, global_id 4902 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 1 -- 172.21.15.12:0/3605041200 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff27400be50 con 0x7ff2780a2360 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 4 set_mon_vals no callback set 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:46.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff26ffff700 10 monclient: dump: 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.405 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 10 monclient: shutdown 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 -- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 msgr2=0x7ff2780a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff2740099b0 tx=0x7ff274009250 comp rx=0 tx=0).stop 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 -- 172.21.15.12:0/3605041200 shutdown_connections 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 --2- 172.21.15.12:0/3605041200 >> v2:172.21.15.153:3300/0 conn(0x7ff278133740 0x7ff278133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff2780a1f90 0x7ff27809f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 --2- 172.21.15.12:0/3605041200 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff2780a2360 0x7ff2780a27b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 -- 172.21.15.12:0/3605041200 >> 172.21.15.12:0/3605041200 conn(0x7ff2780a3b50 msgr2=0x7ff2780a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 -- 172.21.15.12:0/3605041200 shutdown_connections 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.390+0000 7ff27f2df700 1 -- 172.21.15.12:0/3605041200 wait complete. 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.391+0000 7ff27f2df700 10 monclient: build_initial_monmap 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.391+0000 7ff27f2df700 10 monclient: monmap: 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.391+0000 7ff27f2df700 1 Processor -- start 2023-09-29T23:18:46.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.391+0000 7ff27f2df700 1 -- start start 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.391+0000 7ff27f2df700 10 monclient: init 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient: _add_conns ranks=[7,3,2,4,1,0,8,6,5] 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 0x7ff278142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-i con 0x7ff2780a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2780a2360 0x7ff278132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-g con 0x7ff2780a2360 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): picked mon.noname-b con 0x7ff278133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27809e510 con 0x7ff278133740 2023-09-29T23:18:46.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff278131110 con 0x7ff2780a2360 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff278130e30 con 0x7ff2780a1f90 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient(hunting): authenticate will time out at 1435.752109s 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50064/0 (socket says 172.21.15.12:50064) 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 -- 172.21.15.12:0/350323238 learned_addr learned my addr 172.21.15.12:0/350323238 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): get_auth_request con 0x7ff278133740 auth_method 0 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27c87a700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2780a2360 0x7ff278132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d07b700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 0x7ff278142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27c87a700 10 monclient(hunting): get_auth_request con 0x7ff2780a2360 auth_method 0 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27c87a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27c87a700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27c87a700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient(hunting): handle_auth_done global_id 4736 payload 824 2023-09-29T23:18:46.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 -- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 msgr2=0x7ff278142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 0x7ff278142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 -- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2780a2360 msgr2=0x7ff278132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2780a2360 0x7ff278132920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient: _finish_hunting 0 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 monclient: found mon.noname-b 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 -- 172.21.15.12:0/350323238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268009420 con 0x7ff278133740 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d07b700 10 monclient: get_auth_request con 0x7ff2780a1f90 auth_method 0 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d07b700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 0x7ff278142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27d87c700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff27000cef0 tx=0x7ff270007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff27000ae00 con 0x7ff278133740 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: dump: 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.413 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 5 monclient: authenticate success, global_id 4736 2023-09-29T23:18:46.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2700168a0 con 0x7ff278133740 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient: _renew_subs 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2740094f0 con 0x7ff278133740 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff270016e90 con 0x7ff278133740 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26f7fe700 4 set_mon_vals no callback set 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.392+0000 7ff26d7fa700 10 monclient: dump: 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:46.415 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:46.416 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 10 monclient: _renew_subs 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27804c960 con 0x7ff278133740 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff270016a10 con 0x7ff278133740 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278070bc0 con 0x7ff278133740 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff26d7fa700 1 --2- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 0x7ff25c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.393+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff270012070 con 0x7ff278133740 2023-09-29T23:18:46.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.394+0000 7ff27d07b700 1 --2- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 0x7ff25c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:46.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.394+0000 7ff27d07b700 10 monclient: get_auth_request con 0x7ff25c036850 auth_method 0 2023-09-29T23:18:46.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.394+0000 7ff27d07b700 1 --2- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 0x7ff25c038d00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff2680093f0 tx=0x7ff268008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:46.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.398+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff270014920 con 0x7ff278133740 2023-09-29T23:18:46.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.398+0000 7ff26d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:46.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.398+0000 7ff26d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:46.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.592+0000 7ff27f2df700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:46.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.592+0000 7ff27f2df700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:46.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.592+0000 7ff27f2df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:46.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.592+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff278066790 con 0x7ff278133740 2023-09-29T23:18:46.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.593+0000 7ff26d7fa700 1 -- 172.21.15.12:0/350323238 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+9978 (secure 0 0 0) 0x7ff270014330 con 0x7ff278133740 2023-09-29T23:18:46.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.593+0000 7ff26d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-29T23:18:46.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.593+0000 7ff26d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:18:46.593 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:46.593 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"fsid":"b8fec2b7-0020-4de5-aeb3-24af8bd53167","created":"2023-09-29T23:13:26.341576+0000","modified":"2023-09-29T23:13:39.119641+0000","last_up_change":"2023-09-29T23:13:31.402198+0000","last_in_change":"2023-09-29T23:13:28.389101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:13:32.778797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:13:36.755733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"41e52849-9960-40e6-97ad-dc6b6ba4c787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":83761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":83761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":83761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":83761}]},"public_addr":"172.21.15.12:6802/83761","cluster_addr":"172.21.15.12:6804/83761","heartbeat_back_addr":"172.21.15.12:6807/83761","heartbeat_front_addr":"172.21.15.12:6806/83761","state":["exists","up"]},{"osd":1,"uuid":"3765ea7f-51a2-489c-b4ba-c9aab2f2bd60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":83762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6801","nonce":83762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6805","nonce":83762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":83762}]},"public_addr":"172.21.15.12:6800/83762","cluster_addr":"172.21.15.12:6801/83762","heartbeat_back_addr":"172.21.15.12:6805/83762","heartbeat_front_addr":"172.21.15.12:6803/83762","state":["exists","up"]},{"osd":2,"uuid":"c4d5f6b9-7c5d-40b9-9176-0930acb9cd20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":83763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6809","nonce":83763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":83763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":83763}]},"public_addr":"172.21.15.12:6808/83763","cluster_addr":"172.21.15.12:6809/83763","heartbeat_back_addr":"172.21.15.12:6811/83763","heartbeat_front_addr":"172.21.15.12:6810/83763","state":["exists","up"]},{"osd":3,"uuid":"da0e2f13-62a9-4cc0-81a9-e7eb6b9f9b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":83793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":83793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":83793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":83793}]},"public_addr":"172.21.15.153:6800/83793","cluster_addr":"172.21.15.153:6801/83793","heartbeat_back_addr":"172.21.15.153:6804/83793","heartbeat_front_addr":"172.21.15.153:6802/83793","state":["exists","up"]},{"osd":4,"uuid":"558fc6a6-cfca-4177-9514-ff36bb7d87bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":83794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":83794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":83794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":83794}]},"public_addr":"172.21.15.153:6803/83794","cluster_addr":"172.21.15.153:6805/83794","heartbeat_back_addr":"172.21.15.153:6807/83794","heartbeat_front_addr":"172.21.15.153:6806/83794","state":["exists","up"]},{"osd":5,"uuid":"c0b953bb-ccc4-4fc2-a85a-15f012096923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":83795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":83795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":83795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":83795}]},"public_addr":"172.21.15.153:6808/83795","cluster_addr":"172.21.15.153:6809/83795","heartbeat_back_addr":"172.21.15.153:6811/83795","heartbeat_front_addr":"172.21.15.153:6810/83795","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.551171+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.484006+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.491551+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.520272+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.501043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:13:29.509993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:18:46.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 msgr2=0x7ff25c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 0x7ff25c038d00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff2680093f0 tx=0x7ff268008040 comp rx=0 tx=0).stop 2023-09-29T23:18:46.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 10 monclient: shutdown 2023-09-29T23:18:46.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:46.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 msgr2=0x7ff27812d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff27000cef0 tx=0x7ff270007860 comp rx=0 tx=0).stop 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 shutdown_connections 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7ff2780a1f90 0x7ff278142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> v2:172.21.15.153:6812/83257 conn(0x7ff25c036850 0x7ff25c038d00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7ff278133740 0x7ff27812d3f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 --2- 172.21.15.12:0/350323238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2780a2360 0x7ff278132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 >> 172.21.15.12:0/350323238 conn(0x7ff2780a3b50 msgr2=0x7ff27806e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 shutdown_connections 2023-09-29T23:18:46.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:46.595+0000 7ff27f2df700 1 -- 172.21.15.12:0/350323238 wait complete. 2023-09-29T23:18:47.608 INFO:tasks.ceph:Scrubbing osd.0 2023-09-29T23:18:47.608 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 10 monclient: get_monmap_and_config 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 10 monclient: build_initial_monmap 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 10 monclient: monmap: 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:47.744996+0000 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:47.744996+0000 2023-09-29T23:18:47.744 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:47.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 1 Processor -- start 2023-09-29T23:18:47.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 1 -- start start 2023-09-29T23:18:47.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.744+0000 7f68b7311700 10 monclient: init 2023-09-29T23:18:47.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:47.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient: _add_conns ranks=[8,4,10,2,6,7,3,5,9,1,0] 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-e con 0x7f68b00631f0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-c con 0x7f68b009de00 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 0x7f68b0139c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-f con 0x7f68b01378a0 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b009f1f0 con 0x7f68b01378a0 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f68b009add0 con 0x7f68b009de00 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f68b009e710 con 0x7f68b00631f0 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b7311700 10 monclient(hunting): authenticate will time out at 1437.105572s 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55266/0 (socket says 172.21.15.12:55266) 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56600/0 (socket says 172.21.15.12:56600) 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 1 -- 172.21.15.12:0/1282895307 learned_addr learned my addr 172.21.15.12:0/1282895307 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): get_auth_request con 0x7f68b00631f0 auth_method 0 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): get_auth_request con 0x7f68b009de00 auth_method 0 2023-09-29T23:18:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b50ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b58ae700 1 --2- 172.21.15.12:0/1282895307 >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 0x7f68b0139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 10 monclient(hunting): handle_auth_done global_id 4954 payload 274 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 1 -- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 msgr2=0x7f68b009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.745+0000 7f68b48ac700 1 -- 172.21.15.12:0/1282895307 >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 msgr2=0x7f68b0139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 1 --2- 172.21.15.12:0/1282895307 >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 0x7f68b0139c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 10 monclient: _finish_hunting 0 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 1 monclient: found mon.noname-c 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 1 -- 172.21.15.12:0/1282895307 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0068ed0 con 0x7f68b009de00 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b58ae700 10 monclient: get_auth_request con 0x7f68b01378a0 auth_method 0 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b50ad700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b58ae700 1 --2- 172.21.15.12:0/1282895307 >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 0x7f68b0139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b48ac700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098e0 tx=0x7f68a00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7db3d9675e3e979a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 1 -- 172.21.15.12:0/1282895307 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f68a000bac0 con 0x7f68b009de00 2023-09-29T23:18:47.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: dump: 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.759 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: _finish_auth 0 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 1 -- 172.21.15.12:0/1282895307 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a000bc40 con 0x7f68b009de00 2023-09-29T23:18:47.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 5 monclient: authenticate success, global_id 4954 2023-09-29T23:18:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 1 -- 172.21.15.12:0/1282895307 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f68a000bdb0 con 0x7f68b009de00 2023-09-29T23:18:47.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68a7fff700 10 monclient: dump: 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.762 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 4 set_mon_vals no callback set 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 10 monclient: shutdown 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:47.763 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 1 -- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 msgr2=0x7f68b009e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.746+0000 7f68b7311700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098e0 tx=0x7f68a00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 -- 172.21.15.12:0/1282895307 shutdown_connections 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 --2- 172.21.15.12:0/1282895307 >> v2:172.21.15.153:3300/0 conn(0x7f68b01378a0 0x7f68b0139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f68b00631f0 0x7f68b009d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 --2- 172.21.15.12:0/1282895307 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f68b009de00 0x7f68b009e1d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 -- 172.21.15.12:0/1282895307 >> 172.21.15.12:0/1282895307 conn(0x7f68b009fd10 msgr2=0x7f68b00a0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 -- 172.21.15.12:0/1282895307 shutdown_connections 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 1 -- 172.21.15.12:0/1282895307 wait complete. 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 10 monclient: build_initial_monmap 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.747+0000 7f68b7311700 10 monclient: monmap: 2023-09-29T23:18:47.764 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:47.765 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 Processor -- start 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 -- start start 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient: init 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient: _add_conns ranks=[7,1,0,3,4,5,8,6,2] 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f68b00631f0 0x7f68b0097f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-i con 0x7f68b00631f0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68b009de00 0x7f68b00984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-f con 0x7f68b009de00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 0x7f68b00989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): picked mon.noname-a con 0x7f68b01378a0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b0133430 con 0x7f68b01378a0 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b009a820 con 0x7f68b009de00 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f68b009a3c0 con 0x7f68b00631f0 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b7311700 10 monclient(hunting): authenticate will time out at 1437.108806s 2023-09-29T23:18:47.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 0x7f68b00989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 0x7f68b00989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51266/0 (socket says 172.21.15.12:51266) 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 1 -- 172.21.15.12:0/3267214060 learned_addr learned my addr 172.21.15.12:0/3267214060 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 10 monclient(hunting): get_auth_request con 0x7f68b01378a0 auth_method 0 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.748+0000 7f68b58ae700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68b009de00 0x7f68b00984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): get_auth_request con 0x7f68b009de00 auth_method 0 2023-09-29T23:18:47.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b50ad700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f68b00631f0 0x7f68b0097f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b50ad700 10 monclient(hunting): get_auth_request con 0x7f68b00631f0 auth_method 0 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b50ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b50ad700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b50ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b48ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient(hunting): handle_auth_done global_id 5139 payload 824 2023-09-29T23:18:47.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 -- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f68b00631f0 msgr2=0x7f68b0097f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f68b00631f0 0x7f68b0097f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 -- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68b009de00 msgr2=0x7f68b00984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68b009de00 0x7f68b00984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient: _finish_hunting 0 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 monclient: found mon.noname-a 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 -- 172.21.15.12:0/3267214060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac009ae0 con 0x7f68b01378a0 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b58ae700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 0x7f68b00989e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c8b0 tx=0x7f68a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f68a8011a80 con 0x7f68b01378a0 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.770 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: dump: 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.771 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:47.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 5 monclient: authenticate success, global_id 5139 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a8011c20 con 0x7f68b01378a0 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 10 monclient: _renew_subs 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f68a8011d90 con 0x7f68b01378a0 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a0009420 con 0x7f68b01378a0 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a77fe700 4 set_mon_vals no callback set 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68a57fa700 10 monclient: dump: 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.773 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.774 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 10 monclient: _renew_subs 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.749+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b004c960 con 0x7f68b01378a0 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68b7311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68b006a530 con 0x7f68b01378a0 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f68a8011600 con 0x7f68b01378a0 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.153:6812/83257 conn(0x7f6894036850 0x7f6894038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.775 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f68a80471e0 con 0x7f68b01378a0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.12:6802/83761 conn(0x7f689403c2f0 0x7f689403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 --> v2:172.21.15.12:6802/83761 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f689403ed70 con 0x7f689403c2f0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68a8049870 con 0x7f68b01378a0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.750+0000 7f68a57fa700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b50ad700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.153:6812/83257 conn(0x7f6894036850 0x7f6894038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b48ac700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.12:6802/83761 conn(0x7f689403c2f0 0x7f689403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b50ad700 10 monclient: get_auth_request con 0x7f6894036850 auth_method 0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b48ac700 10 monclient: get_auth_request con 0x7f689403c2f0 auth_method 0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b48ac700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.12:6802/83761 conn(0x7f689403c2f0 0x7f689403e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68b50ad700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.153:6812/83257 conn(0x7f6894036850 0x7f6894038d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009ab0 tx=0x7f68ac00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.776 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.751+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== osd.0 v2:172.21.15.12:6802/83761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f689403ed70 con 0x7f689403c2f0 2023-09-29T23:18:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.764+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 --> v2:172.21.15.12:6802/83761 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f68b0064bb0 con 0x7f689403c2f0 2023-09-29T23:18:47.777 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.764+0000 7f68a57fa700 1 -- 172.21.15.12:0/3267214060 <== osd.0 v2:172.21.15.12:6802/83761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f68b0064bb0 con 0x7f689403c2f0 2023-09-29T23:18:47.777 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:47.777 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:47.778 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 >> v2:172.21.15.12:6802/83761 conn(0x7f689403c2f0 msgr2=0x7f689403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.12:6802/83761 conn(0x7f689403c2f0 0x7f689403e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 >> v2:172.21.15.153:6812/83257 conn(0x7f6894036850 msgr2=0x7f6894038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 --2- 172.21.15.12:0/3267214060 >> v2:172.21.15.153:6812/83257 conn(0x7f6894036850 0x7f6894038d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009ab0 tx=0x7f68ac00b040 comp rx=0 tx=0).stop 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 10 monclient: shutdown 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 msgr2=0x7f68b00989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 --2- 172.21.15.12:0/3267214060 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f68b01378a0 0x7f68b00989e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c8b0 tx=0x7f68a8008040 comp rx=0 tx=0).stop 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b58ae700 1 -- 172.21.15.12:0/3267214060 reap_dead start 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 shutdown_connections 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 >> 172.21.15.12:0/3267214060 conn(0x7f68b009fd10 msgr2=0x7f68b0136fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 shutdown_connections 2023-09-29T23:18:47.780 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.765+0000 7f68b7311700 1 -- 172.21.15.12:0/3267214060 wait complete. 2023-09-29T23:18:47.780 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-29T23:18:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.899+0000 7f64c37ca700 10 monclient: get_monmap_and_config 2023-09-29T23:18:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.899+0000 7f64c37ca700 10 monclient: build_initial_monmap 2023-09-29T23:18:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.899+0000 7f64c37ca700 10 monclient: monmap: 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:47.900366+0000 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:47.900366+0000 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:47.901 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 Processor -- start 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 -- start start 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient: init 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient: _add_conns ranks=[7,10,1,3,8,0,6,9,4,2,5] 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a1f90 0x7f64bc09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-i con 0x7f64bc0a1f90 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f64bc0a2360 0x7f64bc0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-f con 0x7f64bc0a2360 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f64bc133740 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc068ed0 con 0x7f64bc133740 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc09e350 con 0x7f64bc0a2360 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc09ef70 con 0x7f64bc0a1f90 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.900+0000 7f64c37ca700 10 monclient(hunting): authenticate will time out at 1437.260753s 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f64bc0a2360 0x7f64bc0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a1f90 0x7f64bc09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1d67700 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f64bc133740 0x7f64bc133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.12:37564/0 (socket says 172.21.15.12:37564) 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1d67700 1 -- 172.21.15.12:0/977777078 learned_addr learned my addr 172.21.15.12:0/977777078 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:47.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): get_auth_request con 0x7f64bc0a2360 auth_method 0 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): get_auth_request con 0x7f64bc0a1f90 auth_method 0 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient(hunting): handle_auth_done global_id 5131 payload 274 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 -- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a1f90 msgr2=0x7f64bc09f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 --2- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a1f90 0x7f64bc09f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 -- 172.21.15.12:0/977777078 >> v1:172.21.15.153:6789/0 conn(0x7f64bc133740 legacy=0x7f64bc133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient: _finish_hunting 0 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 monclient: found mon.noname-f 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 -- 172.21.15.12:0/977777078 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc04c960 con 0x7f64bc0a2360 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c0d65700 1 --2- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0a2360 0x7f64bc0a27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64b00098e0 tx=0x7f64b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e875572f3d8c012a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 1 -- 172.21.15.12:0/977777078 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64b000bac0 con 0x7f64bc0a2360 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: dump: 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.913 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: _finish_auth 0 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c37ca700 5 monclient: authenticate success, global_id 5131 2023-09-29T23:18:47.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 1 -- 172.21.15.12:0/977777078 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b000bc40 con 0x7f64bc0a2360 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c37ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c37ca700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 1 -- 172.21.15.12:0/977777078 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64b000bdb0 con 0x7f64bc0a2360 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64c37ca700 4 set_mon_vals no callback set 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.901+0000 7f64bbfff700 10 monclient: dump: 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.915 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.916 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 10 monclient: shutdown 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 -- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0a2360 msgr2=0x7f64bc0a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 --2- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0a2360 0x7f64bc0a27b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64b00098e0 tx=0x7f64b00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 -- 172.21.15.12:0/977777078 shutdown_connections 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 --2- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a1f90 0x7f64bc09f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 --2- 172.21.15.12:0/977777078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0a2360 0x7f64bc0a27b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 -- 172.21.15.12:0/977777078 >> 172.21.15.12:0/977777078 conn(0x7f64bc0a3b50 msgr2=0x7f64bc0a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 -- 172.21.15.12:0/977777078 shutdown_connections 2023-09-29T23:18:47.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 1 -- 172.21.15.12:0/977777078 wait complete. 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.902+0000 7f64c37ca700 10 monclient: build_initial_monmap 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient: monmap: 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:47.918 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 1 Processor -- start 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 1 -- start start 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient: init 2023-09-29T23:18:47.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient: _add_conns ranks=[3,7,2,8,0,6,1,5,4] 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc0a1f90 0x7f64bc1420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-g con 0x7f64bc0a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 0x7f64bc132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-i con 0x7f64bc0a2360 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.903+0000 7f64c37ca700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): picked mon.noname-b con 0x7f64bc133740 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc142f80 con 0x7f64bc133740 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc09e490 con 0x7f64bc0a1f90 2023-09-29T23:18:47.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc130b50 con 0x7f64bc0a2360 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient(hunting): authenticate will time out at 1437.264019s 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 0x7f64bc132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50074/0 (socket says 172.21.15.12:50074) 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 -- 172.21.15.12:0/1172305940 learned_addr learned my addr 172.21.15.12:0/1172305940 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): get_auth_request con 0x7f64bc133740 auth_method 0 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc0a1f90 0x7f64bc1420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): get_auth_request con 0x7f64bc0a2360 auth_method 0 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): get_auth_request con 0x7f64bc0a1f90 auth_method 0 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1566700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient(hunting): handle_auth_done global_id 4745 payload 824 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 -- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 msgr2=0x7f64bc132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 0x7f64bc132920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 -- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc0a1f90 msgr2=0x7f64bc1420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc0a1f90 0x7f64bc1420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:47.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient: _finish_hunting 0 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 monclient: found mon.noname-b 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 -- 172.21.15.12:0/1172305940 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0009420 con 0x7f64bc133740 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c0d65700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 0x7f64bc132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c1d67700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00cc40 tx=0x7f64ac001a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64ac00ad30 con 0x7f64bc133740 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: dump: 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.925 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: _finish_auth 0 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 5 monclient: authenticate success, global_id 4745 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient: _renew_subs 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac0070f0 con 0x7f64bc133740 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc131110 con 0x7f64bc133740 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:47.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64ac0076e0 con 0x7f64bc133740 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64bb7fe700 4 set_mon_vals no callback set 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64b97fa700 10 monclient: dump: 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:47.927 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient: _renew_subs 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.904+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc12fe20 con 0x7f64bc133740 2023-09-29T23:18:47.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f64ac007860 con 0x7f64bc133740 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64b97fa700 1 --2- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 0x7f64a8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f64ac0475c0 con 0x7f64bc133740 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64c37ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64c37ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64c37ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc070bc0 con 0x7f64bc133740 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.905+0000 7f64c1566700 1 --2- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 0x7f64a8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.906+0000 7f64c1566700 10 monclient: get_auth_request con 0x7f64a8036850 auth_method 0 2023-09-29T23:18:47.929 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.906+0000 7f64c1566700 1 --2- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 0x7f64a8038d00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009d10 tx=0x7f64b40091a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.910+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64ac00a8b0 con 0x7f64bc133740 2023-09-29T23:18:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.910+0000 7f64b97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:47.930 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:47.910+0000 7f64b97fa700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:48.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.101+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f64bc066790 con 0x7f64a8036850 2023-09-29T23:18:48.102 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.102+0000 7f64b97fa700 1 -- 172.21.15.12:0/1172305940 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f64bc066790 con 0x7f64a8036850 2023-09-29T23:18:48.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 msgr2=0x7f64a8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 0x7f64a8038d00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009d10 tx=0x7f64b40091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 10 monclient: shutdown 2023-09-29T23:18:48.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 msgr2=0x7f64bc12d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00cc40 tx=0x7f64ac001a50 comp rx=0 tx=0).stop 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 shutdown_connections 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> v2:172.21.15.153:6812/83257 conn(0x7f64a8036850 0x7f64a8038d00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f64bc133740 0x7f64bc12d4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc0a1f90 0x7f64bc1420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 --2- 172.21.15.12:0/1172305940 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f64bc0a2360 0x7f64bc132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 >> 172.21.15.12:0/1172305940 conn(0x7f64bc0a3b50 msgr2=0x7f64bc06f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 shutdown_connections 2023-09-29T23:18:48.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.104+0000 7f64c37ca700 1 -- 172.21.15.12:0/1172305940 wait complete. 2023-09-29T23:18:48.106 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 0 to deep-scrub 2023-09-29T23:18:48.115 INFO:tasks.ceph:Scrubbing osd.1 2023-09-29T23:18:48.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:48.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.239+0000 7f7754a32700 10 monclient: get_monmap_and_config 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.239+0000 7f7754a32700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.239+0000 7f7754a32700 10 monclient: monmap: 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:48.240546+0000 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:48.240546+0000 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:48.240 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:48.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 1 Processor -- start 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 1 -- start start 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 10 monclient: init 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 10 monclient: _add_conns ranks=[4,7,3,1,2,10,5,0,6,9,8] 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.240+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-c con 0x7f77500a1210 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f77500a15e0 0x7f77500a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-i con 0x7f77500a15e0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:48.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77501339b0 0x7f7750133e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-g con 0x7f77501339b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500a0c70 con 0x7f77501339b0 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f775009ec30 con 0x7f77500a1210 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f775009ef10 con 0x7f77500a15e0 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient(hunting): authenticate will time out at 1437.601082s 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56606/0 (socket says 172.21.15.12:56606) 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 -- 172.21.15.12:0/1209584674 learned_addr learned my addr 172.21.15.12:0/1209584674 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): get_auth_request con 0x7f77500a1210 auth_method 0 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774dd9b700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f77500a15e0 0x7f77500a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774ed9d700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77501339b0 0x7f7750133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774dd9b700 10 monclient(hunting): get_auth_request con 0x7f77500a15e0 auth_method 0 2023-09-29T23:18:48.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774ed9d700 10 monclient(hunting): get_auth_request con 0x7f77501339b0 auth_method 0 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient(hunting): handle_auth_done global_id 4963 payload 274 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 -- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f77500a15e0 msgr2=0x7f77500a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f77500a15e0 0x7f77500a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 -- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77501339b0 msgr2=0x7f7750133e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:48.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77501339b0 0x7f7750133e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 monclient: found mon.noname-c 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 -- 172.21.15.12:0/1209584674 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750068ed0 con 0x7f77500a1210 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774e59c700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7740009f50 tx=0x7f7740009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=936ed182d340fba4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 1 -- 172.21.15.12:0/1209584674 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f774000e070 con 0x7f77500a1210 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: dump: 2023-09-29T23:18:48.255 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.256 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 1 -- 172.21.15.12:0/1209584674 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7740007400 con 0x7f77500a1210 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 5 monclient: authenticate success, global_id 4963 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 1 -- 172.21.15.12:0/1209584674 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f77400092e0 con 0x7f77500a1210 2023-09-29T23:18:48.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f7754a32700 4 set_mon_vals no callback set 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.241+0000 7f774d59a700 10 monclient: dump: 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.258 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 10 monclient: shutdown 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 -- 172.21.15.12:0/1209584674 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 msgr2=0x7f775009f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7740009f50 tx=0x7f7740009840 comp rx=0 tx=0).stop 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 -- 172.21.15.12:0/1209584674 shutdown_connections 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77501339b0 0x7f7750133e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f77500a1210 0x7f775009f8f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 --2- 172.21.15.12:0/1209584674 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f77500a15e0 0x7f77500a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 -- 172.21.15.12:0/1209584674 >> 172.21.15.12:0/1209584674 conn(0x7f77500a3310 msgr2=0x7f77500a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 -- 172.21.15.12:0/1209584674 shutdown_connections 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 1 -- 172.21.15.12:0/1209584674 wait complete. 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.242+0000 7f7754a32700 10 monclient: monmap: 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.260 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.261 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 Processor -- start 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 -- start start 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient: init 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient: _add_conns ranks=[5,1,8,7,3,4,6,0,2] 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77500a1210 0x7f7750143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-h con 0x7f77500a1210 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:48.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77500a15e0 0x7f7750133050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-f con 0x7f77500a15e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 0x7f775012b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): picked mon.noname-e con 0x7f77501339b0 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775012e8f0 con 0x7f77500a15e0 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f775012cee0 con 0x7f77500a1210 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f775009e870 con 0x7f77501339b0 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.243+0000 7f7754a32700 10 monclient(hunting): authenticate will time out at 1437.603934s 2023-09-29T23:18:48.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 0x7f775012b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77500a15e0 0x7f7750133050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77500a1210 0x7f7750143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 0x7f775012b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55268/0 (socket says 172.21.15.12:55268) 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 -- 172.21.15.12:0/3359392227 learned_addr learned my addr 172.21.15.12:0/3359392227 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): get_auth_request con 0x7f77500a15e0 auth_method 0 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): get_auth_request con 0x7f77500a1210 auth_method 0 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): get_auth_request con 0x7f77501339b0 auth_method 0 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient(hunting): handle_auth_done global_id 4814 payload 824 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 -- 172.21.15.12:0/3359392227 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77500a1210 msgr2=0x7f7750143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 --2- 172.21.15.12:0/3359392227 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f77500a1210 0x7f7750143460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 -- 172.21.15.12:0/3359392227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77500a15e0 msgr2=0x7f7750133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 --2- 172.21.15.12:0/3359392227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77500a15e0 0x7f7750133050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 monclient: found mon.noname-e 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 -- 172.21.15.12:0/3359392227 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7740009ae0 con 0x7f77501339b0 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774ed9d700 1 --2- 172.21.15.12:0/3359392227 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 0x7f775012b930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f774800d890 tx=0x7f7748008040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7748007430 con 0x7f77501339b0 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: dump: 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.267 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748012a70 con 0x7f77501339b0 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 5 monclient: authenticate success, global_id 4814 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 10 monclient: _renew_subs 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775004c960 con 0x7f77501339b0 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7748012be0 con 0x7f77501339b0 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f774cd99700 4 set_mon_vals no callback set 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f77467fc700 10 monclient: dump: 2023-09-29T23:18:48.269 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.270 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 10 monclient: _renew_subs 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.244+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775012e250 con 0x7f77501339b0 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f77480125f0 con 0x7f77501339b0 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f7754a32700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7750067060 con 0x7f77501339b0 2023-09-29T23:18:48.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.153:6812/83257 conn(0x7f7730036850 0x7f7730038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7748047560 con 0x7f77501339b0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.12:6800/83762 conn(0x7f773003c2f0 0x7f773003e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 --> v2:172.21.15.12:6800/83762 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f773003ed70 con 0x7f773003c2f0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7748012d60 con 0x7f77501339b0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f77467fc700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f774e59c700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.153:6812/83257 conn(0x7f7730036850 0x7f7730038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.245+0000 7f774e59c700 10 monclient: get_auth_request con 0x7f7730036850 auth_method 0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.246+0000 7f774dd9b700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.12:6800/83762 conn(0x7f773003c2f0 0x7f773003e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.246+0000 7f774dd9b700 10 monclient: get_auth_request con 0x7f773003c2f0 auth_method 0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.246+0000 7f774dd9b700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.12:6800/83762 conn(0x7f773003c2f0 0x7f773003e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.246+0000 7f774e59c700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.153:6812/83257 conn(0x7f7730036850 0x7f7730038d00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7740009f50 tx=0x7f7740018000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.246+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== osd.1 v2:172.21.15.12:6800/83762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f773003ed70 con 0x7f773003c2f0 2023-09-29T23:18:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.259+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 --> v2:172.21.15.12:6800/83762 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7750064bb0 con 0x7f773003c2f0 2023-09-29T23:18:48.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.259+0000 7f77467fc700 1 -- 172.21.15.12:0/3359392227 <== osd.1 v2:172.21.15.12:6800/83762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7750064bb0 con 0x7f773003c2f0 2023-09-29T23:18:48.273 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:48.274 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:48.274 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 >> v2:172.21.15.12:6800/83762 conn(0x7f773003c2f0 msgr2=0x7f773003e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.12:6800/83762 conn(0x7f773003c2f0 0x7f773003e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 >> v2:172.21.15.153:6812/83257 conn(0x7f7730036850 msgr2=0x7f7730038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 --2- 172.21.15.12:0/3359392227 >> v2:172.21.15.153:6812/83257 conn(0x7f7730036850 0x7f7730038d00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7740009f50 tx=0x7f7740018000 comp rx=0 tx=0).stop 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 10 monclient: shutdown 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 msgr2=0x7f775012b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 --2- 172.21.15.12:0/3359392227 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f77501339b0 0x7f775012b930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f774800d890 tx=0x7f7748008040 comp rx=0 tx=0).stop 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f774ed9d700 1 -- 172.21.15.12:0/3359392227 reap_dead start 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 shutdown_connections 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 >> 172.21.15.12:0/3359392227 conn(0x7f77500a3310 msgr2=0x7f7750136110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 shutdown_connections 2023-09-29T23:18:48.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.260+0000 7f7754a32700 1 -- 172.21.15.12:0/3359392227 wait complete. 2023-09-29T23:18:48.277 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-29T23:18:48.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.397+0000 7f8db54cb700 10 monclient: get_monmap_and_config 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.397+0000 7f8db54cb700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.397+0000 7f8db54cb700 10 monclient: monmap: 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:48.398241+0000 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:48.398241+0000 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:48.398 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.397+0000 7f8db54cb700 1 Processor -- start 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 1 -- start start 2023-09-29T23:18:48.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient: init 2023-09-29T23:18:48.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient: _add_conns ranks=[3,0,1,6,7,9,10,8,4,5,2] 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 0x7f8db009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-g con 0x7f8db00a1f90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8db00a2360 addr v1:172.21.15.12:6789/0 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8db0138980 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db0068ed0 con 0x7f8db00a2360 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db004c960 con 0x7f8db0138980 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db009e350 con 0x7f8db00a1f90 2023-09-29T23:18:48.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8db54cb700 10 monclient(hunting): authenticate will time out at 1437.758484s 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 0x7f8db00a27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33420/0 (socket says 172.21.15.12:33420) 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 -- 172.21.15.12:0/1587864557 learned_addr learned my addr 172.21.15.12:0/1587864557 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 1 --2- 172.21.15.12:0/1587864557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 0x7f8db009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907968268 0 0) 0x7f8db0068ed0 con 0x7f8db00a2360 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): get_auth_request con 0x7f8db00a1f90 auth_method 0 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 10 monclient(hunting): my global_id is 5148 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d98003570 con 0x7f8db00a2360 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 -- 172.21.15.12:0/1587864557 >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 legacy=0x7f8db00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 -- 172.21.15.12:0/1587864557 >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 legacy=0x7f8db00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 --1- 172.21.15.12:0/1587864557 >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 0x7f8db00a27b0 :-1 s=OPENED pgs=145 cs=1 l=1).handle_message read tag failed 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dae7fc700 1 --1- 172.21.15.12:0/1587864557 >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 0x7f8db00a27b0 :-1 s=OPENED pgs=145 cs=1 l=1).fault on lossy channel, failing 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.398+0000 7f8dadffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.12:6789/0 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068364067 0 0) 0x7f8db004c960 con 0x7f8db0138980 2023-09-29T23:18:48.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient(hunting): my global_id is 5158 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0068ed0 con 0x7f8db0138980 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 10 monclient(hunting): handle_auth_done global_id 4773 payload 274 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 1 -- 172.21.15.12:0/1587864557 >> v1:172.21.15.153:6789/0 conn(0x7f8db0138980 legacy=0x7f8db013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 1 -- 172.21.15.12:0/1587864557 >> v1:172.21.15.12:6789/0 conn(0x7f8db00a2360 legacy=0x7f8db00a27b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 1 monclient: found mon.noname-g 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.153:3301/0 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 1 -- 172.21.15.12:0/1587864557 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db009da70 con 0x7f8db00a1f90 2023-09-29T23:18:48.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8daeffd700 1 --2- 172.21.15.12:0/1587864557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 0x7f8db009f5e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8db009ff60 tx=0x7f8da00091a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=3cdcb75889a00acf server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 <== mon.3 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8da0011ab0 con 0x7f8db00a1f90 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: dump: 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 <== mon.3 v2:172.21.15.153:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da0011c30 con 0x7f8db00a1f90 2023-09-29T23:18:48.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8db54cb700 5 monclient: authenticate success, global_id 4773 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8db54cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8db54cb700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 1 -- 172.21.15.12:0/1587864557 <== mon.3 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8da0011da0 con 0x7f8db00a1f90 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8db54cb700 4 set_mon_vals no callback set 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.399+0000 7f8dadffb700 10 monclient: dump: 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 10 monclient: shutdown 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 -- 172.21.15.12:0/1587864557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 msgr2=0x7f8db009f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1587864557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 0x7f8db009f5e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8db009ff60 tx=0x7f8da00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 -- 172.21.15.12:0/1587864557 shutdown_connections 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1587864557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8db00a1f90 0x7f8db009f5e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 -- 172.21.15.12:0/1587864557 >> 172.21.15.12:0/1587864557 conn(0x7f8db00a3b50 msgr2=0x7f8db00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 -- 172.21.15.12:0/1587864557 shutdown_connections 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 1 -- 172.21.15.12:0/1587864557 wait complete. 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.400+0000 7f8db54cb700 10 monclient: monmap: 2023-09-29T23:18:48.415 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.416 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 Processor -- start 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 -- start start 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient: init 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient: _add_conns ranks=[2,5,7,0,6,3,8,4,1] 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-b con 0x7f8db00a1f90 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8db00a2360 0x7f8db0097a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-h con 0x7f8db00a2360 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:48.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8db0138980 0x7f8db008c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): picked mon.noname-i con 0x7f8db0138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db008ff10 con 0x7f8db00a1f90 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8db008fc30 con 0x7f8db00a2360 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8db00901f0 con 0x7f8db0138980 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8db54cb700 10 monclient(hunting): authenticate will time out at 1437.761619s 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50088/0 (socket says 172.21.15.12:50088) 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 1 -- 172.21.15.12:0/1466405041 learned_addr learned my addr 172.21.15.12:0/1466405041 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): get_auth_request con 0x7f8db00a1f90 auth_method 0 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8dae7fc700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8db00a2360 0x7f8db0097a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8daf7fe700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8db0138980 0x7f8db008c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8dae7fc700 10 monclient(hunting): get_auth_request con 0x7f8db00a2360 auth_method 0 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8dae7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8dae7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.401+0000 7f8dae7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daf7fe700 10 monclient(hunting): get_auth_request con 0x7f8db0138980 auth_method 0 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 10 monclient(hunting): handle_auth_done global_id 4754 payload 824 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 -- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8db0138980 msgr2=0x7f8db008c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8db0138980 0x7f8db008c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 -- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8db00a2360 msgr2=0x7f8db0097a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8db00a2360 0x7f8db0097a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 monclient: found mon.noname-b 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 -- 172.21.15.12:0/1466405041 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0009420 con 0x7f8db00a1f90 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8daeffd700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8da00030c0 tx=0x7f8da0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8da001bd00 con 0x7f8db00a1f90 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: dump: 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.421 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.422 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da001c8d0 con 0x7f8db00a1f90 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 5 monclient: authenticate success, global_id 4754 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 10 monclient: _renew_subs 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db008ed40 con 0x7f8db00a1f90 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8da00193f0 con 0x7f8db00a1f90 2023-09-29T23:18:48.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8d96ffd700 10 monclient: dump: 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.424 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 10 monclient: _renew_subs 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db0142f70 con 0x7f8db00a1f90 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.402+0000 7f8dad7fa700 4 set_mon_vals no callback set 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8da001c450 con 0x7f8db00a1f90 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8db54cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8db54cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8db54cb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8d96ffd700 1 --2- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 0x7f8d9c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db009e520 con 0x7f8db00a1f90 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8da0048b80 con 0x7f8db00a1f90 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8dae7fc700 1 --2- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 0x7f8d9c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.403+0000 7f8dae7fc700 10 monclient: get_auth_request con 0x7f8d9c036850 auth_method 0 2023-09-29T23:18:48.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.404+0000 7f8dae7fc700 1 --2- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 0x7f8d9c038d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8db006ee90 tx=0x7f8da4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.408+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8da001ca40 con 0x7f8db00a1f90 2023-09-29T23:18:48.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.408+0000 7f8d96ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.408+0000 7f8d96ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:48.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.600+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8db008fda0 con 0x7f8d9c036850 2023-09-29T23:18:48.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.601+0000 7f8d96ffd700 1 -- 172.21.15.12:0/1466405041 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8db008fda0 con 0x7f8d9c036850 2023-09-29T23:18:48.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 msgr2=0x7f8d9c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 0x7f8d9c038d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8db006ee90 tx=0x7f8da4009040 comp rx=0 tx=0).stop 2023-09-29T23:18:48.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 10 monclient: shutdown 2023-09-29T23:18:48.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 msgr2=0x7f8db01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8da00030c0 tx=0x7f8da0016000 comp rx=0 tx=0).stop 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 shutdown_connections 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> v2:172.21.15.153:6812/83257 conn(0x7f8d9c036850 0x7f8d9c038d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8db00a1f90 0x7f8db01421d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f8db0138980 0x7f8db008c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 --2- 172.21.15.12:0/1466405041 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f8db00a2360 0x7f8db0097a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 >> 172.21.15.12:0/1466405041 conn(0x7f8db00a3b50 msgr2=0x7f8db00a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 shutdown_connections 2023-09-29T23:18:48.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.603+0000 7f8db54cb700 1 -- 172.21.15.12:0/1466405041 wait complete. 2023-09-29T23:18:48.605 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 1 to deep-scrub 2023-09-29T23:18:48.615 INFO:tasks.ceph:Scrubbing osd.2 2023-09-29T23:18:48.615 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:48.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.736+0000 7fe58e687700 10 monclient: get_monmap_and_config 2023-09-29T23:18:48.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.736+0000 7fe58e687700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.736+0000 7fe58e687700 10 monclient: monmap: 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:48.737405+0000 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:48.737405+0000 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.737 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:48.738 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 Processor -- start 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 -- start start 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient: init 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient: _add_conns ranks=[1,0,2,4,3,5,8,10,6,7,9] 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe5880a1210 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5880a15e0 addr v1:172.21.15.12:6789/0 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 0x7fe58813bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-b con 0x7fe588139c00 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe588068ed0 con 0x7fe5880a15e0 2023-09-29T23:18:48.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58804c960 con 0x7fe5880a1210 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5880a0c70 con 0x7fe588139c00 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58e687700 10 monclient(hunting): authenticate will time out at 1438.097781s 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe5877fe700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 0x7fe5880a1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33426/0 (socket says 172.21.15.12:33426) 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe5877fe700 1 -- 172.21.15.12:0/1967683712 learned_addr learned my addr 172.21.15.12:0/1967683712 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58cc24700 1 --2- 172.21.15.12:0/1967683712 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 0x7fe58813bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58cc24700 10 monclient(hunting): get_auth_request con 0x7fe588139c00 auth_method 0 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58cc24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58cc24700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe58cc24700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.737+0000 7fe5877fe700 1 -- 172.21.15.12:0/1967683712 >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 legacy=0x7fe5880a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe5877fe700 1 -- 172.21.15.12:0/1967683712 >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 legacy=0x7fe5880a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe5877fe700 1 --1- 172.21.15.12:0/1967683712 >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 0x7fe5880a1a10 :-1 s=OPENED pgs=146 cs=1 l=1).handle_message read tag failed 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe5877fe700 1 --1- 172.21.15.12:0/1967683712 >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 0x7fe5880a1a10 :-1 s=OPENED pgs=146 cs=1 l=1).fault on lossy channel, failing 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.12:6789/0 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 1 -- 172.21.15.12:0/1967683712 >> v1:172.21.15.153:6789/0 conn(0x7fe5880a1210 legacy=0x7fe58809f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 1 -- 172.21.15.12:0/1967683712 >> v1:172.21.15.12:6789/0 conn(0x7fe5880a15e0 legacy=0x7fe5880a1a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 1 monclient: found mon.noname-b 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 1 -- 172.21.15.12:0/1967683712 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5880a0000 con 0x7fe588139c00 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58cc24700 1 --2- 172.21.15.12:0/1967683712 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 0x7fe58813bfe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe588066920 tx=0x7fe580009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d6f743f2ada7e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 1 -- 172.21.15.12:0/1967683712 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe58000e070 con 0x7fe588139c00 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: dump: 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.750 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.751 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58e687700 5 monclient: authenticate success, global_id 4763 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58e687700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 1 -- 172.21.15.12:0/1967683712 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58000bc30 con 0x7fe588139c00 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58e687700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe58e687700 4 set_mon_vals no callback set 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 1 -- 172.21.15.12:0/1967683712 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe58000bda0 con 0x7fe588139c00 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.738+0000 7fe586ffd700 10 monclient: dump: 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.753 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.754 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 10 monclient: shutdown 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 -- 172.21.15.12:0/1967683712 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 msgr2=0x7fe58813bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 --2- 172.21.15.12:0/1967683712 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 0x7fe58813bfe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe588066920 tx=0x7fe580009780 comp rx=0 tx=0).stop 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 -- 172.21.15.12:0/1967683712 shutdown_connections 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 --2- 172.21.15.12:0/1967683712 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe588139c00 0x7fe58813bfe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 -- 172.21.15.12:0/1967683712 >> 172.21.15.12:0/1967683712 conn(0x7fe5880a3310 msgr2=0x7fe5880a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 -- 172.21.15.12:0/1967683712 shutdown_connections 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 1 -- 172.21.15.12:0/1967683712 wait complete. 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.739+0000 7fe58e687700 10 monclient: monmap: 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.756 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 Processor -- start 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 -- start start 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient: init 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient: _add_conns ranks=[8,3,0,4,7,6,2,1,5] 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 0x7fe58806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-e con 0x7fe5880a1210 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:48.757 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5880a15e0 0x7fe58806c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-g con 0x7fe5880a15e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe588139c00 0x7fe58806cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): picked mon.noname-a con 0x7fe588139c00 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58806f860 con 0x7fe588139c00 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58806e730 con 0x7fe5880a15e0 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fe58809e870 con 0x7fe5880a1210 2023-09-29T23:18:48.758 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58e687700 10 monclient(hunting): authenticate will time out at 1438.100670s 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 0x7fe58806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 0x7fe58806c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55284/0 (socket says 172.21.15.12:55284) 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe588139c00 0x7fe58806cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 1 -- 172.21.15.12:0/3722068342 learned_addr learned my addr 172.21.15.12:0/3722068342 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): get_auth_request con 0x7fe5880a1210 auth_method 0 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): get_auth_request con 0x7fe588139c00 auth_method 0 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe587fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe5877fe700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5880a15e0 0x7fe58806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe58cc24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe5877fe700 10 monclient(hunting): get_auth_request con 0x7fe5880a15e0 auth_method 0 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe5877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe5877fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.740+0000 7fe5877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 -- 172.21.15.12:0/3722068342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5880a15e0 msgr2=0x7fe58806c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5880a15e0 0x7fe58806c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 -- 172.21.15.12:0/3722068342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe588139c00 msgr2=0x7fe58806cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe588139c00 0x7fe58806cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 monclient: found mon.noname-e 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 -- 172.21.15.12:0/3722068342 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580009a20 con 0x7fe5880a1210 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe5877fe700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5880a15e0 0x7fe58806c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe587fff700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 0x7fe58806c130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe57800c590 tx=0x7fe578007f30 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe578001080 con 0x7fe5880a1210 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: dump: 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.762 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.763 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 5 monclient: authenticate success, global_id 4823 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe578011a10 con 0x7fe5880a1210 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe5780102e0 con 0x7fe5880a1210 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 10 monclient: _renew_subs 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58806dd90 con 0x7fe5880a1210 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.764 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe5867fc700 4 set_mon_vals no callback set 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe56ffff700 10 monclient: dump: 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 10 monclient: _renew_subs 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.766 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.741+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58806e900 con 0x7fe5880a1210 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe578011b80 con 0x7fe5880a1210 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe58e687700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe58806eaf0 con 0x7fe5880a1210 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.153:6812/83257 conn(0x7fe574036850 0x7fe574038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe5780462d0 con 0x7fe5880a1210 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.12:6808/83763 conn(0x7fe57403c2f0 0x7fe57403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 --> v2:172.21.15.12:6808/83763 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe57403ed70 con 0x7fe57403c2f0 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe578011e30 con 0x7fe5880a1210 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe56ffff700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:48.767 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe58cc24700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.12:6808/83763 conn(0x7fe57403c2f0 0x7fe57403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe58cc24700 10 monclient: get_auth_request con 0x7fe57403c2f0 auth_method 0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe5877fe700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.153:6812/83257 conn(0x7fe574036850 0x7fe574038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe5877fe700 10 monclient: get_auth_request con 0x7fe574036850 auth_method 0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.742+0000 7fe58cc24700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.12:6808/83763 conn(0x7fe57403c2f0 0x7fe57403e700 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.743+0000 7fe5877fe700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.153:6812/83257 conn(0x7fe574036850 0x7fe574038d00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0010b0 tx=0x7fe57c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.768 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.743+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== osd.2 v2:172.21.15.12:6808/83763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7fe57403ed70 con 0x7fe57403c2f0 2023-09-29T23:18:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 --> v2:172.21.15.12:6808/83763 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe58812c410 con 0x7fe57403c2f0 2023-09-29T23:18:48.769 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe56ffff700 1 -- 172.21.15.12:0/3722068342 <== osd.2 v2:172.21.15.12:6808/83763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe58812c410 con 0x7fe57403c2f0 2023-09-29T23:18:48.769 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:48.769 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:48.769 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:48.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 >> v2:172.21.15.12:6808/83763 conn(0x7fe57403c2f0 msgr2=0x7fe57403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.12:6808/83763 conn(0x7fe57403c2f0 0x7fe57403e700 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 >> v2:172.21.15.153:6812/83257 conn(0x7fe574036850 msgr2=0x7fe574038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 --2- 172.21.15.12:0/3722068342 >> v2:172.21.15.153:6812/83257 conn(0x7fe574036850 0x7fe574038d00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0010b0 tx=0x7fe57c009040 comp rx=0 tx=0).stop 2023-09-29T23:18:48.771 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 10 monclient: shutdown 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 msgr2=0x7fe58806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 --2- 172.21.15.12:0/3722068342 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fe5880a1210 0x7fe58806c130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe57800c590 tx=0x7fe578007f30 comp rx=0 tx=0).stop 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58cc24700 1 -- 172.21.15.12:0/3722068342 reap_dead start 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 shutdown_connections 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 >> 172.21.15.12:0/3722068342 conn(0x7fe5880a3310 msgr2=0x7fe588135d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 shutdown_connections 2023-09-29T23:18:48.772 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.756+0000 7fe58e687700 1 -- 172.21.15.12:0/3722068342 wait complete. 2023-09-29T23:18:48.772 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-29T23:18:48.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.891+0000 7f5a5bab4700 10 monclient: get_monmap_and_config 2023-09-29T23:18:48.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.891+0000 7f5a5bab4700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.891+0000 7f5a5bab4700 10 monclient: monmap: 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:48.892361+0000 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:48.892361+0000 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.892 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:48.893 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 Processor -- start 2023-09-29T23:18:48.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 -- start start 2023-09-29T23:18:48.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient: init 2023-09-29T23:18:48.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient: _add_conns ranks=[4,1,3,10,7,6,9,5,2,0,8] 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-c con 0x7f5a540a1f90 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5a540a2360 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a54138980 0x7f5a5413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-g con 0x7f5a54138980 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a54068ed0 con 0x7f5a540a2360 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5409e350 con 0x7f5a54138980 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5409ef70 con 0x7f5a540a1f90 2023-09-29T23:18:48.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.892+0000 7f5a5bab4700 10 monclient(hunting): authenticate will time out at 1438.252987s 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56610/0 (socket says 172.21.15.12:56610) 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 -- 172.21.15.12:0/3753705692 learned_addr learned my addr 172.21.15.12:0/3753705692 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): get_auth_request con 0x7f5a540a1f90 auth_method 0 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a54138980 0x7f5a5413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): get_auth_request con 0x7f5a54138980 auth_method 0 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient(hunting): handle_auth_done global_id 4972 payload 274 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 -- 172.21.15.12:0/3753705692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a54138980 msgr2=0x7f5a5413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a54138980 0x7f5a5413ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 -- 172.21.15.12:0/3753705692 >> v1:172.21.15.153:6789/0 conn(0x7f5a540a2360 legacy=0x7f5a540a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 monclient: found mon.noname-c 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 -- 172.21.15.12:0/3753705692 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5404c960 con 0x7f5a540a1f90 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a59850700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009f50 tx=0x7f5a50009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b2c8458f297f0ce9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 1 -- 172.21.15.12:0/3753705692 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a5000e070 con 0x7f5a540a1f90 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: dump: 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.908 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5bab4700 5 monclient: authenticate success, global_id 4972 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 1 -- 172.21.15.12:0/3753705692 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5000be80 con 0x7f5a540a1f90 2023-09-29T23:18:48.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5bab4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5bab4700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 1 -- 172.21.15.12:0/3753705692 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a500092e0 con 0x7f5a540a1f90 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5bab4700 4 set_mon_vals no callback set 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:48.910 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.893+0000 7f5a5884e700 10 monclient: dump: 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.911 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.912 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 10 monclient: shutdown 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3753705692 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 msgr2=0x7f5a5409f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009f50 tx=0x7f5a50009840 comp rx=0 tx=0).stop 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3753705692 shutdown_connections 2023-09-29T23:18:48.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a54138980 0x7f5a5413ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3753705692 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f5a540a1f90 0x7f5a5409f5e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3753705692 >> 172.21.15.12:0/3753705692 conn(0x7f5a540a3b50 msgr2=0x7f5a540a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3753705692 shutdown_connections 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3753705692 wait complete. 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 10 monclient: build_initial_monmap 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.894+0000 7f5a5bab4700 10 monclient: monmap: 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:48.914 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 Processor -- start 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 -- start start 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient: init 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient: _add_conns ranks=[5,6,7,1,4,3,0,8,2] 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-h con 0x7f5a540a1f90 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-d con 0x7f5a540a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5a54138980 0x7f5a54097f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): picked mon.noname-i con 0x7f5a54138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540a0030 con 0x7f5a540a1f90 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540901b0 con 0x7f5a540a2360 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5408ee00 con 0x7f5a54138980 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5bab4700 10 monclient(hunting): authenticate will time out at 1438.255629s 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37124/0 (socket says 172.21.15.12:37124) 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3302/0 says I am v2:172.21.15.12:45550/0 (socket says 172.21.15.12:45550) 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 1 -- 172.21.15.12:0/3306594183 learned_addr learned my addr 172.21.15.12:0/3306594183 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): get_auth_request con 0x7f5a540a2360 auth_method 0 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): get_auth_request con 0x7f5a540a1f90 auth_method 0 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.918 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5a54138980 0x7f5a54097f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): get_auth_request con 0x7f5a54138980 auth_method 0 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5904f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a59850700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.895+0000 7f5a5a051700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 10 monclient(hunting): handle_auth_done global_id 4911 payload 824 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 -- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5a54138980 msgr2=0x7f5a54097f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5a54138980 0x7f5a54097f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 -- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 msgr2=0x7f5a541421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:48.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 10 monclient: _finish_hunting 0 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 monclient: found mon.noname-d 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 -- 172.21.15.12:0/3306594183 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a50009ae0 con 0x7f5a540a2360 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a59850700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5904f700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c7f0 tx=0x7f5a44007f50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a44001120 con 0x7f5a540a2360 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: dump: 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.921 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.922 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a44011a10 con 0x7f5a540a2360 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 5 monclient: authenticate success, global_id 4911 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 10 monclient: _renew_subs 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:48.923 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5408fed0 con 0x7f5a540a2360 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a440102e0 con 0x7f5a540a2360 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a49ffb700 10 monclient: dump: 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:48.924 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:48.925 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a4bfff700 4 set_mon_vals no callback set 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 10 monclient: _renew_subs 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.896+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5408e0c0 con 0x7f5a540a2360 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5a44011b80 con 0x7f5a540a2360 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a49ffb700 1 --2- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 0x7f5a38038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5a44046270 con 0x7f5a540a2360 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a5bab4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a5bab4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a5bab4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a54070bc0 con 0x7f5a540a2360 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a59850700 1 --2- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 0x7f5a38038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.897+0000 7f5a59850700 10 monclient: get_auth_request con 0x7f5a38036850 auth_method 0 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.898+0000 7f5a59850700 1 --2- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 0x7f5a38038d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009f50 tx=0x7f5a50018000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.902+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a44017020 con 0x7f5a540a2360 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.902+0000 7f5a49ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:48.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:48.902+0000 7f5a49ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:49.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.092+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a54143120 con 0x7f5a38036850 2023-09-29T23:18:49.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.092+0000 7f5a49ffb700 1 -- 172.21.15.12:0/3306594183 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5a54143120 con 0x7f5a38036850 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.094+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 msgr2=0x7f5a38038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.094+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 0x7f5a38038d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009f50 tx=0x7f5a50018000 comp rx=0 tx=0).stop 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 10 monclient: shutdown 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 msgr2=0x7f5a54097a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c7f0 tx=0x7f5a44007f50 comp rx=0 tx=0).stop 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 shutdown_connections 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> v2:172.21.15.153:6812/83257 conn(0x7f5a38036850 0x7f5a38038d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f5a54138980 0x7f5a54097f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f5a540a1f90 0x7f5a541421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 --2- 172.21.15.12:0/3306594183 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f5a540a2360 0x7f5a54097a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 >> 172.21.15.12:0/3306594183 conn(0x7f5a540a3b50 msgr2=0x7f5a5409a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 shutdown_connections 2023-09-29T23:18:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.095+0000 7f5a5bab4700 1 -- 172.21.15.12:0/3306594183 wait complete. 2023-09-29T23:18:49.098 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 2 to deep-scrub 2023-09-29T23:18:49.106 INFO:tasks.ceph:Scrubbing osd.3 2023-09-29T23:18:49.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.230+0000 7f8a11f43700 10 monclient: get_monmap_and_config 2023-09-29T23:18:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.230+0000 7f8a11f43700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.230+0000 7f8a11f43700 10 monclient: monmap: 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:49.231415+0000 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:49.231415+0000 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:49.232 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 Processor -- start 2023-09-29T23:18:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 -- start start 2023-09-29T23:18:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient: init 2023-09-29T23:18:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient: _add_conns ranks=[10,6,4,8,3,1,0,7,2,5,9] 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8a0c0a1210 0x7f8a0c09f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-f con 0x7f8a0c0a1210 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 0x7f8a0c0a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-d con 0x7f8a0c0a15e0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-c con 0x7f8a0c1339b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c0a0c70 con 0x7f8a0c0a1210 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c09ec30 con 0x7f8a0c1339b0 2023-09-29T23:18:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c09ef10 con 0x7f8a0c0a15e0 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a11f43700 10 monclient(hunting): authenticate will time out at 1438.591780s 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0affd700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 0x7f8a0c0a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56616/0 (socket says 172.21.15.12:56616) 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 1 -- 172.21.15.12:0/199109159 learned_addr learned my addr 172.21.15.12:0/199109159 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 10 monclient(hunting): get_auth_request con 0x7f8a0c1339b0 auth_method 0 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0affd700 10 monclient(hunting): get_auth_request con 0x7f8a0c0a15e0 auth_method 0 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.231+0000 7f8a0affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 1 --2- 172.21.15.12:0/199109159 >> v2:172.21.15.153:3300/0 conn(0x7f8a0c0a1210 0x7f8a0c09f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): get_auth_request con 0x7f8a0c0a1210 auth_method 0 2023-09-29T23:18:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_done global_id 4981 payload 274 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 -- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 msgr2=0x7f8a0c0a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 0x7f8a0c0a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 -- 172.21.15.12:0/199109159 >> v2:172.21.15.153:3300/0 conn(0x7f8a0c0a1210 msgr2=0x7f8a0c09f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/199109159 >> v2:172.21.15.153:3300/0 conn(0x7f8a0c0a1210 0x7f8a0c09f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 monclient: found mon.noname-c 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 -- 172.21.15.12:0/199109159 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c068ed0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0affd700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 0x7f8a0c0a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009f50 tx=0x7f8a04009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c2fe4167f8fa7891 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 1 -- 172.21.15.12:0/199109159 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8a0400e070 con 0x7f8a0c1339b0 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: dump: 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 1 -- 172.21.15.12:0/199109159 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0400bd90 con 0x7f8a0c1339b0 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a11f43700 5 monclient: authenticate success, global_id 4981 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a11f43700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a11f43700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 1 -- 172.21.15.12:0/199109159 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8a040092e0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a11f43700 4 set_mon_vals no callback set 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.232+0000 7f8a0a7fc700 10 monclient: dump: 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 10 monclient: shutdown 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 -- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 msgr2=0x7f8a0c133e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009f50 tx=0x7f8a04009840 comp rx=0 tx=0).stop 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 -- 172.21.15.12:0/199109159 shutdown_connections 2023-09-29T23:18:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c133e00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 --2- 172.21.15.12:0/199109159 >> v2:172.21.15.153:3300/0 conn(0x7f8a0c0a1210 0x7f8a0c09f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 --2- 172.21.15.12:0/199109159 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f8a0c0a15e0 0x7f8a0c0a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 -- 172.21.15.12:0/199109159 >> 172.21.15.12:0/199109159 conn(0x7f8a0c0a3310 msgr2=0x7f8a0c0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 -- 172.21.15.12:0/199109159 shutdown_connections 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 1 -- 172.21.15.12:0/199109159 wait complete. 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.233+0000 7f8a11f43700 10 monclient: monmap: 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 Processor -- start 2023-09-29T23:18:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 -- start start 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient: init 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient: _add_conns ranks=[3,1,4,0,6,5,2,7,8] 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a0c0a1210 0x7f8a0c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-g con 0x7f8a0c0a1210 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a0c0a15e0 0x7f8a0c132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-f con 0x7f8a0c0a15e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c12b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): picked mon.noname-c con 0x7f8a0c1339b0 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c09e8a0 con 0x7f8a0c0a15e0 2023-09-29T23:18:49.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c1433d0 con 0x7f8a0c0a1210 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c12dfb0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a11f43700 10 monclient(hunting): authenticate will time out at 1438.594663s 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c12b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c12b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56630/0 (socket says 172.21.15.12:56630) 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 1 -- 172.21.15.12:0/3660482875 learned_addr learned my addr 172.21.15.12:0/3660482875 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): get_auth_request con 0x7f8a0c1339b0 auth_method 0 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0affd700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a0c0a15e0 0x7f8a0c132d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 10 monclient(hunting): handle_auth_done global_id 4990 payload 824 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0bfff700 1 -- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a0c0a1210 msgr2=0x7f8a0c06ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.234+0000 7f8a0b7fe700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a0c0a1210 0x7f8a0c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a0c0a1210 0x7f8a0c06ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 -- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a0c0a15e0 msgr2=0x7f8a0c132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a0c0a15e0 0x7f8a0c132d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 monclient: found mon.noname-c 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 -- 172.21.15.12:0/3660482875 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f4009420 con 0x7f8a0c1339b0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0affd700 10 monclient: get_auth_request con 0x7f8a0c0a15e0 auth_method 0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0b7fe700 10 monclient: get_auth_request con 0x7f8a0c0a1210 auth_method 0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0b7fe700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a0c0a1210 0x7f8a0c06ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0affd700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a0c0a15e0 0x7f8a0c132d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a0bfff700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c12b930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c09ed70 tx=0x7f8a04018000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8a04003300 con 0x7f8a0c1339b0 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: dump: 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.257 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a040092e0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 5 monclient: authenticate success, global_id 4990 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 10 monclient: _renew_subs 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a04009ae0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8a040268f0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a037fe700 10 monclient: dump: 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.259 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a09ffb700 4 set_mon_vals no callback set 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 10 monclient: _renew_subs 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.235+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c04c960 con 0x7f8a0c1339b0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8a04023070 con 0x7f8a0c1339b0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a11f43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a0c067060 con 0x7f8a0c1339b0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6812/83257 conn(0x7f89ec036850 0x7f89ec038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8a04049040 con 0x7f8a0c1339b0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6800/83793 conn(0x7f89ec03c180 0x7f89ec03e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 --> v2:172.21.15.153:6800/83793 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89ec03ec00 con 0x7f89ec03c180 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a0404b6d0 con 0x7f8a0c1339b0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a037fe700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a0b7fe700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6812/83257 conn(0x7f89ec036850 0x7f89ec038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a0b7fe700 10 monclient: get_auth_request con 0x7f89ec036850 auth_method 0 2023-09-29T23:18:49.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a0affd700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6800/83793 conn(0x7f89ec03c180 0x7f89ec03e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.236+0000 7f8a0affd700 10 monclient: get_auth_request con 0x7f89ec03c180 auth_method 0 2023-09-29T23:18:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.237+0000 7f8a0b7fe700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6812/83257 conn(0x7f89ec036850 0x7f89ec038d00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89fc006f00 tx=0x7f89fc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.237+0000 7f8a0affd700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6800/83793 conn(0x7f89ec03c180 0x7f89ec03e590 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.237+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== osd.3 v2:172.21.15.153:6800/83793 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f89ec03ec00 con 0x7f89ec03c180 2023-09-29T23:18:49.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.250+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 --> v2:172.21.15.153:6800/83793 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8a0c12e140 con 0x7f89ec03c180 2023-09-29T23:18:49.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.250+0000 7f8a037fe700 1 -- 172.21.15.12:0/3660482875 <== osd.3 v2:172.21.15.153:6800/83793 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8a0c12e140 con 0x7f89ec03c180 2023-09-29T23:18:49.263 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:49.263 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:49.263 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:49.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.250+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6800/83793 conn(0x7f89ec03c180 msgr2=0x7f89ec03e590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.250+0000 7f8a11f43700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6800/83793 conn(0x7f89ec03c180 0x7f89ec03e590 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6812/83257 conn(0x7f89ec036850 msgr2=0x7f89ec038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 --2- 172.21.15.12:0/3660482875 >> v2:172.21.15.153:6812/83257 conn(0x7f89ec036850 0x7f89ec038d00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89fc006f00 tx=0x7f89fc008040 comp rx=0 tx=0).stop 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 10 monclient: shutdown 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 msgr2=0x7f8a0c12b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 --2- 172.21.15.12:0/3660482875 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f8a0c1339b0 0x7f8a0c12b930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c09ed70 tx=0x7f8a04018000 comp rx=0 tx=0).stop 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a0bfff700 1 -- 172.21.15.12:0/3660482875 reap_dead start 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 shutdown_connections 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 >> 172.21.15.12:0/3660482875 conn(0x7f8a0c0a3310 msgr2=0x7f8a0c0a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 shutdown_connections 2023-09-29T23:18:49.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.251+0000 7f8a11f43700 1 -- 172.21.15.12:0/3660482875 wait complete. 2023-09-29T23:18:49.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-29T23:18:49.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.385+0000 7f7942d6d700 10 monclient: get_monmap_and_config 2023-09-29T23:18:49.385 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.385+0000 7f7942d6d700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.385+0000 7f7942d6d700 10 monclient: monmap: 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:49.386555+0000 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:49.386555+0000 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:49.386 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:49.387 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 1 Processor -- start 2023-09-29T23:18:49.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 1 -- start start 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 10 monclient: init 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 10 monclient: _add_conns ranks=[9,5,8,7,10,4,3,1,0,6,2] 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f793c133130 0x7f793c0a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-a con 0x7f793c133130 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.386+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-h con 0x7f793c133500 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 0x7f793c1300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-e con 0x7f793c12fc20 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793c0a27a0 con 0x7f793c133130 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f793c09adf0 con 0x7f793c133500 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f793c09bd60 con 0x7f793c12fc20 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient(hunting): authenticate will time out at 1438.747044s 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f793c133130 0x7f793c0a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 0x7f793c1300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f793c133130 0x7f793c0a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51284/0 (socket says 172.21.15.12:51284) 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 -- 172.21.15.12:0/2079835412 learned_addr learned my addr 172.21.15.12:0/2079835412 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): get_auth_request con 0x7f793c133130 auth_method 0 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): get_auth_request con 0x7f793c12fc20 auth_method 0 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient(hunting): handle_auth_done global_id 5166 payload 274 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 -- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 msgr2=0x7f793c1300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 0x7f793c1300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 -- 172.21.15.12:0/2079835412 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 msgr2=0x7f793c133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 monclient: found mon.noname-a 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:49.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 -- 172.21.15.12:0/2079835412 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c068ed0 con 0x7f793c133130 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f794256c700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 0x7f793c1300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f793c133130 0x7f793c0a0e80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79380099e0 tx=0x7f7938009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56f0346a0c7c35b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 1 -- 172.21.15.12:0/2079835412 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f793800bc40 con 0x7f793c133130 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: dump: 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.401 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 5 monclient: authenticate success, global_id 5166 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 1 -- 172.21.15.12:0/2079835412 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793800bdc0 con 0x7f793c133130 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 1 -- 172.21.15.12:0/2079835412 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f79380118a0 con 0x7f793c133130 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7942d6d700 4 set_mon_vals no callback set 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.387+0000 7f7940d69700 10 monclient: dump: 2023-09-29T23:18:49.402 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.403 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 10 monclient: shutdown 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 -- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f793c133130 msgr2=0x7f793c0a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f793c133130 0x7f793c0a0e80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79380099e0 tx=0x7f7938009250 comp rx=0 tx=0).stop 2023-09-29T23:18:49.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 -- 172.21.15.12:0/2079835412 shutdown_connections 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f793c12fc20 0x7f793c1300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f793c133130 0x7f793c0a0e80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2079835412 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 -- 172.21.15.12:0/2079835412 >> 172.21.15.12:0/2079835412 conn(0x7f793c0a6490 msgr2=0x7f793c0a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 -- 172.21.15.12:0/2079835412 shutdown_connections 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 1 -- 172.21.15.12:0/2079835412 wait complete. 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.388+0000 7f7942d6d700 10 monclient: monmap: 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:49.405 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.406 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 Processor -- start 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 -- start start 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient: init 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient: _add_conns ranks=[4,7,5,1,6,0,3,8,2] 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-c con 0x7f793c12fc20 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f793c133130 0x7f793c142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-i con 0x7f793c133130 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:49.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): picked mon.noname-h con 0x7f793c133500 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f793c11dae0 con 0x7f793c12fc20 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f793c06f510 con 0x7f793c133500 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f793c09b3e0 con 0x7f793c133130 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7942d6d700 10 monclient(hunting): authenticate will time out at 1438.749860s 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56634/0 (socket says 172.21.15.12:56634) 2023-09-29T23:18:49.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 1 -- 172.21.15.12:0/2283350098 learned_addr learned my addr 172.21.15.12:0/2283350098 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 10 monclient(hunting): get_auth_request con 0x7f793c12fc20 auth_method 0 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.389+0000 7f7941d6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient(hunting): handle_auth_done global_id 4999 payload 824 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f794256c700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 -- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f793c133130 msgr2=0x7f793c142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f793c133130 0x7f793c142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 -- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 msgr2=0x7f793c06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c06d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 monclient: found mon.noname-c 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 -- 172.21.15.12:0/2283350098 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934009a20 con 0x7f793c12fc20 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7941d6b700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7938009880 tx=0x7f7938017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7938011db0 con 0x7f793c12fc20 2023-09-29T23:18:49.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: dump: 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.411 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 5 monclient: authenticate success, global_id 4999 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793801f070 con 0x7f793c12fc20 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 10 monclient: _renew_subs 2023-09-29T23:18:49.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79380094f0 con 0x7f793c12fc20 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7938026d80 con 0x7f793c12fc20 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7931ffb700 10 monclient: dump: 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.413 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7933fff700 4 set_mon_vals no callback set 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 10 monclient: _renew_subs 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.390+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793c04c960 con 0x7f793c12fc20 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f7938026480 con 0x7f793c12fc20 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7942d6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7942d6d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7942d6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793c070bc0 con 0x7f793c12fc20 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7931ffb700 1 --2- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 0x7f7920038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.391+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7938021070 con 0x7f793c12fc20 2023-09-29T23:18:49.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.392+0000 7f794156a700 1 --2- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 0x7f7920038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.392+0000 7f794156a700 10 monclient: get_auth_request con 0x7f7920036850 auth_method 0 2023-09-29T23:18:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.392+0000 7f794156a700 1 --2- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 0x7f7920038d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f793c06ed50 tx=0x7f792c006d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.396+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mon.4 v2:172.21.15.12:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79380259d0 con 0x7f793c12fc20 2023-09-29T23:18:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.396+0000 7f7931ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.396+0000 7f7931ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:49.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.589+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f793c11e000 con 0x7f7920036850 2023-09-29T23:18:49.590 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.590+0000 7f7931ffb700 1 -- 172.21.15.12:0/2283350098 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f792c0113f0 con 0x7f7920036850 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 msgr2=0x7f7920038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 0x7f7920038d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f793c06ed50 tx=0x7f792c006d10 comp rx=0 tx=0).stop 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 10 monclient: shutdown 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 msgr2=0x7f793c142790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7938009880 tx=0x7f7938017000 comp rx=0 tx=0).stop 2023-09-29T23:18:49.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 shutdown_connections 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> v2:172.21.15.153:6812/83257 conn(0x7f7920036850 0x7f7920038d00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f793c133500 0x7f793c06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f793c12fc20 0x7f793c142790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 --2- 172.21.15.12:0/2283350098 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f793c133130 0x7f793c142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 >> 172.21.15.12:0/2283350098 conn(0x7f793c0a6490 msgr2=0x7f793c1349d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 shutdown_connections 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.592+0000 7f7942d6d700 1 -- 172.21.15.12:0/2283350098 wait complete. 2023-09-29T23:18:49.594 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 3 to deep-scrub 2023-09-29T23:18:49.604 INFO:tasks.ceph:Scrubbing osd.4 2023-09-29T23:18:49.604 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:49.725 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.725+0000 7f0a2e5c9700 10 monclient: get_monmap_and_config 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.725+0000 7f0a2e5c9700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.725+0000 7f0a2e5c9700 10 monclient: monmap: 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:49.726668+0000 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:49.726668+0000 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:49.726 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:49.727 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.726+0000 7f0a2e5c9700 1 Processor -- start 2023-09-29T23:18:49.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.726+0000 7f0a2e5c9700 1 -- start start 2023-09-29T23:18:49.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.726+0000 7f0a2e5c9700 10 monclient: init 2023-09-29T23:18:49.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient: _add_conns ranks=[0,6,10,4,8,5,9,3,2,1,7] 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a280631f0 addr v1:172.21.15.12:6789/0 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 0x7f0a281388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-d con 0x7f0a2809dfd0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 0x7f0a2813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-f con 0x7f0a2809e3a0 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a28068ed0 con 0x7f0a280631f0 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2809f1f0 con 0x7f0a2809e3a0 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2809add0 con 0x7f0a2809dfd0 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient(hunting): authenticate will time out at 1439.087251s 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 0x7f0a281388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a27fff700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f0a280631f0 0x7f0a2809d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:33430/0 (socket says 172.21.15.12:33430) 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a27fff700 1 -- 172.21.15.12:0/2072073346 learned_addr learned my addr 172.21.15.12:0/2072073346 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): get_auth_request con 0x7f0a2809dfd0 auth_method 0 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient(hunting): handle_auth_done global_id 4929 payload 274 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 -- 172.21.15.12:0/2072073346 >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 msgr2=0x7f0a2813ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/2072073346 >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 0x7f0a2813ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 1 -- 172.21.15.12:0/2072073346 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910337921 0 0) 0x7f0a28068ed0 con 0x7f0a280631f0 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 --2- 172.21.15.12:0/2072073346 >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 0x7f0a2813ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 -- 172.21.15.12:0/2072073346 >> v1:172.21.15.12:6789/0 conn(0x7f0a280631f0 legacy=0x7f0a2809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 monclient: found mon.noname-d 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 -- 172.21.15.12:0/2072073346 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2804c960 con 0x7f0a2809dfd0 2023-09-29T23:18:49.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2cb66700 10 monclient: get_auth_request con 0x7f0a2809e3a0 auth_method 0 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/2072073346 >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 0x7f0a2813ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a277fe700 1 --2- 172.21.15.12:0/2072073346 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 0x7f0a281388b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009910 tx=0x7f0a1c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6e48d853a7c2fc01 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 1 -- 172.21.15.12:0/2072073346 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a1c00bca0 con 0x7f0a2809dfd0 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: dump: 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.740 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 1 -- 172.21.15.12:0/2072073346 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c00be20 con 0x7f0a2809dfd0 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 5 monclient: authenticate success, global_id 4929 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a2e5c9700 4 set_mon_vals no callback set 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 1 -- 172.21.15.12:0/2072073346 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a1c0118a0 con 0x7f0a2809dfd0 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.727+0000 7f0a26ffd700 10 monclient: dump: 2023-09-29T23:18:49.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.743 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 10 monclient: shutdown 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/2072073346 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 msgr2=0x7f0a281388b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/2072073346 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 0x7f0a281388b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009910 tx=0x7f0a1c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/2072073346 shutdown_connections 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/2072073346 >> v2:172.21.15.153:3300/0 conn(0x7f0a2809e3a0 0x7f0a2813ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/2072073346 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f0a2809dfd0 0x7f0a281388b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/2072073346 >> 172.21.15.12:0/2072073346 conn(0x7f0a2809fd10 msgr2=0x7f0a280a0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/2072073346 shutdown_connections 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/2072073346 wait complete. 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.728+0000 7f0a2e5c9700 10 monclient: monmap: 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:49.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 Processor -- start 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 -- start start 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient: init 2023-09-29T23:18:49.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient: _add_conns ranks=[0,2,3,7,4,6,1,5,8] 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 0x7f0a2806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-a con 0x7f0a280631f0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 0x7f0a281328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-b con 0x7f0a2809dfd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a2809e3a0 0x7f0a2811d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): picked mon.noname-g con 0x7f0a2809e3a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2809a820 con 0x7f0a280631f0 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2809a3c0 con 0x7f0a2809dfd0 2023-09-29T23:18:49.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2811fd60 con 0x7f0a2809e3a0 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.729+0000 7f0a2e5c9700 10 monclient(hunting): authenticate will time out at 1439.089992s 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 0x7f0a2806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 0x7f0a281328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 0x7f0a2806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51298/0 (socket says 172.21.15.12:51298) 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 0x7f0a281328d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50106/0 (socket says 172.21.15.12:50106) 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 -- 172.21.15.12:0/1942210024 learned_addr learned my addr 172.21.15.12:0/1942210024 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): get_auth_request con 0x7f0a280631f0 auth_method 0 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): get_auth_request con 0x7f0a2809dfd0 auth_method 0 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a2809e3a0 0x7f0a2811d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient(hunting): handle_auth_done global_id 5184 payload 824 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 -- 172.21.15.12:0/1942210024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a2809e3a0 msgr2=0x7f0a2811d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a2809e3a0 0x7f0a2811d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 -- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 msgr2=0x7f0a281328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 0x7f0a281328d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 monclient: found mon.noname-a 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 -- 172.21.15.12:0/1942210024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c009420 con 0x7f0a280631f0 2023-09-29T23:18:49.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2cb66700 10 monclient: get_auth_request con 0x7f0a2809e3a0 auth_method 0 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a2809e3a0 0x7f0a2811d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a277fe700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0a2809dfd0 0x7f0a281328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a27fff700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 0x7f0a2806d8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800c590 tx=0x7f0a18000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a180070d0 con 0x7f0a280631f0 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: dump: 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.751 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 5 monclient: authenticate success, global_id 5184 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a18007250 con 0x7f0a280631f0 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a18011360 con 0x7f0a280631f0 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a267fc700 4 set_mon_vals no callback set 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a0ffff700 10 monclient: dump: 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.753 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.754 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 10 monclient: _renew_subs 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2811f9f0 con 0x7f0a280631f0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 10 monclient: _renew_subs 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.730+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2806e480 con 0x7f0a280631f0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.731+0000 7f0a2e5c9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.731+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a28067060 con 0x7f0a280631f0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.731+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0a18005b30 con 0x7f0a280631f0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.731+0000 7f0a0ffff700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6812/83257 conn(0x7f0a14036850 0x7f0a14038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0a18046310 con 0x7f0a280631f0 2023-09-29T23:18:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a0ffff700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6803/83794 conn(0x7f0a1403c2f0 0x7f0a1403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 --> v2:172.21.15.153:6803/83794 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a1403ed70 con 0x7f0a1403c2f0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a180489a0 con 0x7f0a280631f0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a0ffff700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a277fe700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6812/83257 conn(0x7f0a14036850 0x7f0a14038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a277fe700 10 monclient: get_auth_request con 0x7f0a14036850 auth_method 0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6803/83794 conn(0x7f0a1403c2f0 0x7f0a1403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a2cb66700 10 monclient: get_auth_request con 0x7f0a1403c2f0 auth_method 0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a277fe700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6812/83257 conn(0x7f0a14036850 0x7f0a14038d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009860 tx=0x7f0a1c017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.732+0000 7f0a2cb66700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6803/83794 conn(0x7f0a1403c2f0 0x7f0a1403e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.733+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== osd.4 v2:172.21.15.153:6803/83794 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f0a1403ed70 con 0x7f0a1403c2f0 2023-09-29T23:18:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.746+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 --> v2:172.21.15.153:6803/83794 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0a28064bb0 con 0x7f0a1403c2f0 2023-09-29T23:18:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.746+0000 7f0a0ffff700 1 -- 172.21.15.12:0/1942210024 <== osd.4 v2:172.21.15.153:6803/83794 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0a28064bb0 con 0x7f0a1403c2f0 2023-09-29T23:18:49.759 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:49.759 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:49.759 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.746+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6803/83794 conn(0x7f0a1403c2f0 msgr2=0x7f0a1403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.746+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6803/83794 conn(0x7f0a1403c2f0 0x7f0a1403e700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6812/83257 conn(0x7f0a14036850 msgr2=0x7f0a14038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/1942210024 >> v2:172.21.15.153:6812/83257 conn(0x7f0a14036850 0x7f0a14038d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009860 tx=0x7f0a1c017000 comp rx=0 tx=0).stop 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 10 monclient: shutdown 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 msgr2=0x7f0a2806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 --2- 172.21.15.12:0/1942210024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a280631f0 0x7f0a2806d8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800c590 tx=0x7f0a18000fe0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2cb66700 1 -- 172.21.15.12:0/1942210024 reap_dead start 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 shutdown_connections 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 >> 172.21.15.12:0/1942210024 conn(0x7f0a2809fd10 msgr2=0x7f0a280a2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.761 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 shutdown_connections 2023-09-29T23:18:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.747+0000 7f0a2e5c9700 1 -- 172.21.15.12:0/1942210024 wait complete. 2023-09-29T23:18:49.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-29T23:18:49.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.880+0000 7fbcdc139700 10 monclient: get_monmap_and_config 2023-09-29T23:18:49.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.880+0000 7fbcdc139700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.881 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.881+0000 7fbcdc139700 10 monclient: monmap: 2023-09-29T23:18:49.881 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.881 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:49.881826+0000 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:49.881826+0000 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.882 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.881+0000 7fbcdc139700 1 Processor -- start 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.881+0000 7fbcdc139700 1 -- start start 2023-09-29T23:18:49.883 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.881+0000 7fbcdc139700 10 monclient: init 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient: _add_conns ranks=[1,4,2,0,3,5,9,10,8,7,6] 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbcd40a1f90 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-c con 0x7fbcd40a2360 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-b con 0x7fbcd4138980 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd4068ed0 con 0x7fbcd40a1f90 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd409e350 con 0x7fbcd4138980 2023-09-29T23:18:49.891 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd409fcf0 con 0x7fbcd40a2360 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcdc139700 10 monclient(hunting): authenticate will time out at 1439.242256s 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50114/0 (socket says 172.21.15.12:50114) 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56636/0 (socket says 172.21.15.12:56636) 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 1 -- 172.21.15.12:0/799252645 learned_addr learned my addr 172.21.15.12:0/799252645 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): get_auth_request con 0x7fbcd4138980 auth_method 0 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.892 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): get_auth_request con 0x7fbcd40a2360 auth_method 0 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.893 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_done global_id 5008 payload 274 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 -- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 msgr2=0x7fbcd413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 -- 172.21.15.12:0/799252645 >> v1:172.21.15.153:6789/0 conn(0x7fbcd40a1f90 legacy=0x7fbcd409f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 monclient: found mon.noname-c 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 -- 172.21.15.12:0/799252645 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd404c960 con 0x7fbcd40a2360 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcda6d6700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.882+0000 7fbcd96d4700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009910 tx=0x7fbcd00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=250799e117f14752 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 1 -- 172.21.15.12:0/799252645 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbcd000bb00 con 0x7fbcd40a2360 2023-09-29T23:18:49.894 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: dump: 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.895 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 5 monclient: authenticate success, global_id 5008 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 1 -- 172.21.15.12:0/799252645 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd000bc80 con 0x7fbcd40a2360 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.896 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 1 -- 172.21.15.12:0/799252645 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbcd000bdf0 con 0x7fbcd40a2360 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 4 set_mon_vals no callback set 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcd8ed3700 10 monclient: dump: 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.897 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.898 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 10 monclient: shutdown 2023-09-29T23:18:49.899 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 -- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 msgr2=0x7fbcd40a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009910 tx=0x7fbcd00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 -- 172.21.15.12:0/799252645 shutdown_connections 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fbcd4138980 0x7fbcd413ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 --2- 172.21.15.12:0/799252645 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fbcd40a2360 0x7fbcd40a27b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 -- 172.21.15.12:0/799252645 >> 172.21.15.12:0/799252645 conn(0x7fbcd40a3b50 msgr2=0x7fbcd40a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 -- 172.21.15.12:0/799252645 shutdown_connections 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.883+0000 7fbcdc139700 1 -- 172.21.15.12:0/799252645 wait complete. 2023-09-29T23:18:49.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 10 monclient: build_initial_monmap 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 10 monclient: monmap: 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:49.901 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 1 Processor -- start 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 1 -- start start 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 10 monclient: init 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.884+0000 7fbcdc139700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient: _add_conns ranks=[6,8,7,1,0,5,3,4,2] 2023-09-29T23:18:49.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-d con 0x7fbcd40a1f90 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 0x7fbcd408d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-e con 0x7fbcd40a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbcd4138980 0x7fbcd408d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): picked mon.noname-i con 0x7fbcd4138980 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4097650 con 0x7fbcd40a1f90 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4097f70 con 0x7fbcd4138980 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd409ffc0 con 0x7fbcd40a2360 2023-09-29T23:18:49.903 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient(hunting): authenticate will time out at 1439.245107s 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 0x7fbcd408d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37152/0 (socket says 172.21.15.12:37152) 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 -- 172.21.15.12:0/296893844 learned_addr learned my addr 172.21.15.12:0/296893844 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): get_auth_request con 0x7fbcd40a1f90 auth_method 0 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): get_auth_request con 0x7fbcd40a2360 auth_method 0 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.904 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbcd4138980 0x7fbcd408d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): get_auth_request con 0x7fbcd4138980 auth_method 0 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:49.905 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcda6d6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient(hunting): handle_auth_done global_id 4938 payload 824 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 -- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 msgr2=0x7fbcd408d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 0x7fbcd408d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 -- 172.21.15.12:0/296893844 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbcd4138980 msgr2=0x7fbcd408d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbcd4138980 0x7fbcd408d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient: _finish_hunting 0 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 monclient: found mon.noname-d 2023-09-29T23:18:49.906 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 -- 172.21.15.12:0/296893844 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc009ae0 con 0x7fbcd40a1f90 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd96d4700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 0x7fbcd408d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcd9ed5700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c590 tx=0x7fbcc400c8a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc4001120 con 0x7fbcd40a1f90 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: dump: 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.907 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.908 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 5 monclient: authenticate success, global_id 4938 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4011a10 con 0x7fbcd40a1f90 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient: _renew_subs 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc40102e0 con 0x7fbcd40a1f90 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd0009420 con 0x7fbcd40a1f90 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbccbfff700 4 set_mon_vals no callback set 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:49.909 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.885+0000 7fbcc9ffb700 10 monclient: dump: 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:49.910 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 10 monclient: _renew_subs 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd40906a0 con 0x7fbcd40a1f90 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbcc4011b80 con 0x7fbcd40a1f90 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcc9ffb700 1 --2- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 0x7fbcb8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbcc4046320 con 0x7fbcd40a1f90 2023-09-29T23:18:49.911 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.886+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd409b670 con 0x7fbcd40a1f90 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.887+0000 7fbcd96d4700 1 --2- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 0x7fbcb8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.887+0000 7fbcd96d4700 10 monclient: get_auth_request con 0x7fbcb8036880 auth_method 0 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.887+0000 7fbcd96d4700 1 --2- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 0x7fbcb8038d30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcd406b460 tx=0x7fbcd0017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.891+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcc4017070 con 0x7fbcd40a1f90 2023-09-29T23:18:49.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.891+0000 7fbcc9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:49.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:49.891+0000 7fbcc9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:50.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.086+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcd4070bc0 con 0x7fbcb8036880 2023-09-29T23:18:50.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.086+0000 7fbcc9ffb700 1 -- 172.21.15.12:0/296893844 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbcd4070bc0 con 0x7fbcb8036880 2023-09-29T23:18:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 msgr2=0x7fbcb8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 0x7fbcb8038d30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcd406b460 tx=0x7fbcd0017000 comp rx=0 tx=0).stop 2023-09-29T23:18:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 10 monclient: shutdown 2023-09-29T23:18:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 msgr2=0x7fbcd408cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c590 tx=0x7fbcc400c8a0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 shutdown_connections 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> v2:172.21.15.153:6812/83257 conn(0x7fbcb8036880 0x7fbcb8038d30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fbcd4138980 0x7fbcd408d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7fbcd40a1f90 0x7fbcd408cb20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 --2- 172.21.15.12:0/296893844 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7fbcd40a2360 0x7fbcd408d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 >> 172.21.15.12:0/296893844 conn(0x7fbcd40a3b50 msgr2=0x7fbcd409a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 shutdown_connections 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.088+0000 7fbcdc139700 1 -- 172.21.15.12:0/296893844 wait complete. 2023-09-29T23:18:50.090 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 4 to deep-scrub 2023-09-29T23:18:50.100 INFO:tasks.ceph:Scrubbing osd.5 2023-09-29T23:18:50.100 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-29T23:18:50.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.223+0000 7f9e8d340700 10 monclient: get_monmap_and_config 2023-09-29T23:18:50.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.223+0000 7f9e8d340700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.223+0000 7f9e8d340700 10 monclient: monmap: 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:50.224373+0000 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:50.224373+0000 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.224 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:50.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 Processor -- start 2023-09-29T23:18:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 -- start start 2023-09-29T23:18:50.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient: init 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient: _add_conns ranks=[10,9,1,8,3,2,7,4,0,6,5] 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9e880a1210 0x7f9e8809f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-f con 0x7f9e880a1210 addr v2:172.21.15.153:3300/0 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e880a15e0 0x7f9e880a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-a con 0x7f9e880a15e0 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9e881339b0 addr v1:172.21.15.153:6789/0 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e88068ed0 con 0x7f9e881339b0 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e880a0c70 con 0x7f9e880a15e0 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8809ec30 con 0x7f9e880a1210 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e8d340700 10 monclient(hunting): authenticate will time out at 1439.584738s 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e880a15e0 0x7f9e880a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7f9e880a15e0 0x7f9e880a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51306/0 (socket says 172.21.15.12:51306) 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 1 -- 172.21.15.12:0/1890711854 learned_addr learned my addr 172.21.15.12:0/1890711854 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): get_auth_request con 0x7f9e880a15e0 auth_method 0 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.224+0000 7f9e86ffd700 1 --2- 172.21.15.12:0/1890711854 >> v2:172.21.15.153:3300/0 conn(0x7f9e880a1210 0x7f9e8809f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e86ffd700 10 monclient(hunting): get_auth_request con 0x7f9e880a1210 auth_method 0 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e86ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e86ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e86ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 10 monclient(hunting): handle_auth_done global_id 5193 payload 274 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 -- 172.21.15.12:0/1890711854 >> v2:172.21.15.153:3300/0 conn(0x7f9e880a1210 msgr2=0x7f9e8809f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 --2- 172.21.15.12:0/1890711854 >> v2:172.21.15.153:3300/0 conn(0x7f9e880a1210 0x7f9e8809f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 -- 172.21.15.12:0/1890711854 >> v1:172.21.15.153:6789/0 conn(0x7f9e881339b0 legacy=0x7f9e88133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 monclient: found mon.noname-a 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 -- 172.21.15.12:0/1890711854 --> v2:172.21.15.12:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8804c960 con 0x7f9e880a15e0 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e867fc700 1 --2- 172.21.15.12:0/1890711854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e880a15e0 0x7f9e880a1a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e88066920 tx=0x7f9e7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7cf25bf6269646 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 1 -- 172.21.15.12:0/1890711854 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9e7c00bce0 con 0x7f9e880a15e0 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: dump: 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.238 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.239 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e8d340700 5 monclient: authenticate success, global_id 5193 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 1 -- 172.21.15.12:0/1890711854 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c00be60 con 0x7f9e880a15e0 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e8d340700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e8d340700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e8d340700 4 set_mon_vals no callback set 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 1 -- 172.21.15.12:0/1890711854 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9e7c0118a0 con 0x7f9e880a15e0 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.225+0000 7f9e85ffb700 10 monclient: dump: 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.241 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 10 monclient: shutdown 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 -- 172.21.15.12:0/1890711854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e880a15e0 msgr2=0x7f9e880a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 --2- 172.21.15.12:0/1890711854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e880a15e0 0x7f9e880a1a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e88066920 tx=0x7f9e7c009250 comp rx=0 tx=0).stop 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 -- 172.21.15.12:0/1890711854 shutdown_connections 2023-09-29T23:18:50.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 --2- 172.21.15.12:0/1890711854 >> v2:172.21.15.153:3300/0 conn(0x7f9e880a1210 0x7f9e8809f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 --2- 172.21.15.12:0/1890711854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e880a15e0 0x7f9e880a1a10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 -- 172.21.15.12:0/1890711854 >> 172.21.15.12:0/1890711854 conn(0x7f9e880a3310 msgr2=0x7f9e880a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 -- 172.21.15.12:0/1890711854 shutdown_connections 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 1 -- 172.21.15.12:0/1890711854 wait complete. 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.226+0000 7f9e8d340700 10 monclient: monmap: 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:50.243 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 Processor -- start 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 -- start start 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient: init 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient: _add_conns ranks=[3,7,6,4,8,2,0,5,1] 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880a1210 0x7f9e8812be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-g con 0x7f9e880a1210 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9e880a15e0 0x7f9e8806eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-i con 0x7f9e880a15e0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 0x7f9e88132c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): picked mon.noname-d con 0x7f9e881339b0 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8806f670 con 0x7f9e880a1210 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8812e120 con 0x7f9e881339b0 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8812cb30 con 0x7f9e880a15e0 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e8d340700 10 monclient(hunting): authenticate will time out at 1439.587583s 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 0x7f9e88132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9e880a15e0 0x7f9e8806eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880a1210 0x7f9e8812be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 0x7f9e88132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37166/0 (socket says 172.21.15.12:37166) 2023-09-29T23:18:50.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 1 -- 172.21.15.12:0/88399271 learned_addr learned my addr 172.21.15.12:0/88399271 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): get_auth_request con 0x7f9e881339b0 auth_method 0 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): get_auth_request con 0x7f9e880a15e0 auth_method 0 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): get_auth_request con 0x7f9e880a1210 auth_method 0 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.227+0000 7f9e86ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 10 monclient(hunting): handle_auth_done global_id 4947 payload 824 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 -- 172.21.15.12:0/88399271 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9e880a15e0 msgr2=0x7f9e8806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 --2- 172.21.15.12:0/88399271 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7f9e880a15e0 0x7f9e8806eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 -- 172.21.15.12:0/88399271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880a1210 msgr2=0x7f9e8812be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 --2- 172.21.15.12:0/88399271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880a1210 0x7f9e8812be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 monclient: found mon.noname-d 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 -- 172.21.15.12:0/88399271 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0094f0 con 0x7f9e881339b0 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e877fe700 1 --2- 172.21.15.12:0/88399271 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 0x7f9e88132c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000c8e0 tx=0x7f9e80001710 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9e80012ab0 con 0x7f9e881339b0 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:50.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: dump: 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.250 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e80012c50 con 0x7f9e881339b0 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 5 monclient: authenticate success, global_id 4947 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 10 monclient: _renew_subs 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8812dea0 con 0x7f9e881339b0 2023-09-29T23:18:50.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9e80012dc0 con 0x7f9e881339b0 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e6effd700 10 monclient: dump: 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.252 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e857fa700 4 set_mon_vals no callback set 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 10 monclient: _renew_subs 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.228+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8812cfe0 con 0x7f9e881339b0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e8d340700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.12:3303/0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e880a5e40 con 0x7f9e881339b0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9e80003030 con 0x7f9e881339b0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6812/83257 conn(0x7f9e74036850 0x7f9e74038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9e80047160 con 0x7f9e881339b0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6808/83795 conn(0x7f9e7403c2f0 0x7f9e7403e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 --> v2:172.21.15.153:6808/83795 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e7403ed70 con 0x7f9e7403c2f0 2023-09-29T23:18:50.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e86ffd700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6812/83257 conn(0x7f9e74036850 0x7f9e74038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== mon.6 v2:172.21.15.12:3303/0 6 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e800497f0 con 0x7f9e881339b0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e6effd700 10 monclient: handle_get_version_reply finishing 1 version 18 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e86ffd700 10 monclient: get_auth_request con 0x7f9e74036850 auth_method 0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e867fc700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6808/83795 conn(0x7f9e7403c2f0 0x7f9e7403e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e867fc700 10 monclient: get_auth_request con 0x7f9e7403c2f0 auth_method 0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.229+0000 7f9e86ffd700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6812/83257 conn(0x7f9e74036850 0x7f9e74038d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e78006fd0 tx=0x7f9e78008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.230+0000 7f9e867fc700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6808/83795 conn(0x7f9e7403c2f0 0x7f9e7403e700 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.230+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== osd.5 v2:172.21.15.153:6808/83795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22340 (crc 0 0 0) 0x7f9e7403ed70 con 0x7f9e7403c2f0 2023-09-29T23:18:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.243+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 --> v2:172.21.15.153:6808/83795 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9e8812e290 con 0x7f9e7403c2f0 2023-09-29T23:18:50.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.243+0000 7f9e6effd700 1 -- 172.21.15.12:0/88399271 <== osd.5 v2:172.21.15.153:6808/83795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9e8812e290 con 0x7f9e7403c2f0 2023-09-29T23:18:50.256 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-09-29T23:18:50.256 INFO:teuthology.orchestra.run.smithi012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-29T23:18:50.256 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.243+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6808/83795 conn(0x7f9e7403c2f0 msgr2=0x7f9e7403e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.243+0000 7f9e8d340700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6808/83795 conn(0x7f9e7403c2f0 0x7f9e7403e700 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6812/83257 conn(0x7f9e74036850 msgr2=0x7f9e74038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 --2- 172.21.15.12:0/88399271 >> v2:172.21.15.153:6812/83257 conn(0x7f9e74036850 0x7f9e74038d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9e78006fd0 tx=0x7f9e78008040 comp rx=0 tx=0).stop 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 10 monclient: shutdown 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 msgr2=0x7f9e88132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 --2- 172.21.15.12:0/88399271 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7f9e881339b0 0x7f9e88132c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000c8e0 tx=0x7f9e80001710 comp rx=0 tx=0).stop 2023-09-29T23:18:50.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e877fe700 1 -- 172.21.15.12:0/88399271 reap_dead start 2023-09-29T23:18:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 shutdown_connections 2023-09-29T23:18:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 >> 172.21.15.12:0/88399271 conn(0x7f9e880a3310 msgr2=0x7f9e88135f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 shutdown_connections 2023-09-29T23:18:50.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.244+0000 7f9e8d340700 1 -- 172.21.15.12:0/88399271 wait complete. 2023-09-29T23:18:50.259 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 10 monclient: get_monmap_and_config 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 10 monclient: monmap: 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.379 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:50.380135+0000 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:50.380135+0000 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:50.381 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:50.381 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 1 Processor -- start 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 1 -- start start 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.379+0000 7f6429342700 10 monclient: init 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient: _add_conns ranks=[5,8,4,2,3,1,9,0,10,6,7] 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 0x7f642406f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.389 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): picked mon.noname-h con 0x7f642406f050 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): picked mon.noname-e con 0x7f642409e500 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): picked mon.noname-c con 0x7f642409e950 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6424066a20 con 0x7f642409e950 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f642409f370 con 0x7f642406f050 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f642406f960 con 0x7f642409e500 2023-09-29T23:18:50.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6429342700 10 monclient(hunting): authenticate will time out at 1439.740367s 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55312/0 (socket says 172.21.15.12:55312) 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 -- 172.21.15.12:0/872412901 learned_addr learned my addr 172.21.15.12:0/872412901 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): get_auth_request con 0x7f642409e500 auth_method 0 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:56652/0 (socket says 172.21.15.12:56652) 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): get_auth_request con 0x7f642409e950 auth_method 0 2023-09-29T23:18:50.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 0x7f642406f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 10 monclient(hunting): get_auth_request con 0x7f642406f050 auth_method 0 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient(hunting): handle_auth_done global_id 4850 payload 274 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 -- 172.21.15.12:0/872412901 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 msgr2=0x7f642406f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 0x7f642406f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 -- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 msgr2=0x7f6424133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 monclient: found mon.noname-e 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:18:50.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64227fc700 1 -- 172.21.15.12:0/872412901 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424068ed0 con 0x7f642409e500 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f64237fe700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.380+0000 7f6422ffd700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 0x7f642406f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f64227fc700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f640c009a10 tx=0x7f640c009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=808200822bd8faf0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 1 -- 172.21.15.12:0/872412901 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f640c00bca0 con 0x7f642409e500 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: dump: 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.394 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 5 monclient: authenticate success, global_id 4850 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 1 -- 172.21.15.12:0/872412901 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00be20 con 0x7f642409e500 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 4 set_mon_vals no callback set 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 1 -- 172.21.15.12:0/872412901 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f640c0118a0 con 0x7f642409e500 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6421ffb700 10 monclient: dump: 2023-09-29T23:18:50.396 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.397 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 10 monclient: shutdown 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 -- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 msgr2=0x7f64241332e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f640c009a10 tx=0x7f640c009250 comp rx=0 tx=0).stop 2023-09-29T23:18:50.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 -- 172.21.15.12:0/872412901 shutdown_connections 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f642409e950 0x7f6424133820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7f642406f050 0x7f642406f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 --2- 172.21.15.12:0/872412901 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e500 0x7f64241332e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 -- 172.21.15.12:0/872412901 >> 172.21.15.12:0/872412901 conn(0x7f64240a3330 msgr2=0x7f64240a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 -- 172.21.15.12:0/872412901 shutdown_connections 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.381+0000 7f6429342700 1 -- 172.21.15.12:0/872412901 wait complete. 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.382+0000 7f6429342700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.382+0000 7f6429342700 10 monclient: monmap: 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:50.399 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.400 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.382+0000 7f6429342700 1 Processor -- start 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.382+0000 7f6429342700 1 -- start start 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.382+0000 7f6429342700 10 monclient: init 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient: _add_conns ranks=[0,2,8,4,3,5,1,6,7] 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642406f050 0x7f6424133130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): picked mon.noname-a con 0x7f642406f050 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): picked mon.noname-b con 0x7f642409e500 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 0x7f6424120060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): picked mon.noname-e con 0x7f642409e950 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:50.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424143290 con 0x7f642406f050 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6424121f80 con 0x7f642409e500 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f642409ef60 con 0x7f642409e950 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6429342700 10 monclient(hunting): authenticate will time out at 1439.743275s 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642406f050 0x7f6424133130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50128/0 (socket says 172.21.15.12:50128) 2023-09-29T23:18:50.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 -- 172.21.15.12:0/4038112488 learned_addr learned my addr 172.21.15.12:0/4038112488 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): get_auth_request con 0x7f642409e500 auth_method 0 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): get_auth_request con 0x7f642406f050 auth_method 0 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 0x7f6424120060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): get_auth_request con 0x7f642409e950 auth_method 0 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f6422ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient(hunting): handle_auth_done global_id 4790 payload 824 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 -- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 msgr2=0x7f6424120060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 0x7f6424120060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 -- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642406f050 msgr2=0x7f6424133130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642406f050 0x7f6424133130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 monclient: found mon.noname-b 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.12:3301/0 2023-09-29T23:18:50.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 -- 172.21.15.12:0/4038112488 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414009a20 con 0x7f642409e500 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64237fe700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 0x7f6424120060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.383+0000 7f64227fc700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f640c009a10 tx=0x7f640c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f640c011c90 con 0x7f642409e500 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: dump: 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.406 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 5 monclient: authenticate success, global_id 4790 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c01f070 con 0x7f642409e500 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f640c011e30 con 0x7f642409e500 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f64217fa700 4 set_mon_vals no callback set 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f641affd700 10 monclient: dump: 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.408 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.409 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 10 monclient: _renew_subs 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c0094f0 con 0x7f642409e500 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 10 monclient: _renew_subs 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.384+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642404c960 con 0x7f642409e500 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f640c01a430 con 0x7f642409e500 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6429342700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6429342700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6429342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.12:3301/0 2023-09-29T23:18:50.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642406a410 con 0x7f642409e500 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f641affd700 1 --2- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 0x7f6404038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f640c021070 con 0x7f642409e500 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6422ffd700 1 --2- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 0x7f6404038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.385+0000 7f6422ffd700 10 monclient: get_auth_request con 0x7f6404036780 auth_method 0 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.386+0000 7f6422ffd700 1 --2- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 0x7f6404038c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f641c006ea0 tx=0x7f641c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.389+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f640c01aa60 con 0x7f642409e500 2023-09-29T23:18:50.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.389+0000 7f641affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.389+0000 7f641affd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:50.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.582+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f64241237a0 con 0x7f6404036780 2023-09-29T23:18:50.582 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.582+0000 7f641affd700 1 -- 172.21.15.12:0/4038112488 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f641c005a00 con 0x7f6404036780 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 msgr2=0x7f6404038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 0x7f6404038c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f641c006ea0 tx=0x7f641c008040 comp rx=0 tx=0).stop 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 10 monclient: shutdown 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 msgr2=0x7f642411fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f640c009a10 tx=0x7f640c017000 comp rx=0 tx=0).stop 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 shutdown_connections 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f642409e950 0x7f6424120060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f642406f050 0x7f6424133130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> v2:172.21.15.153:6812/83257 conn(0x7f6404036780 0x7f6404038c30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 --2- 172.21.15.12:0/4038112488 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f642409e500 0x7f642411fb20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 >> 172.21.15.12:0/4038112488 conn(0x7f64240a3330 msgr2=0x7f642409c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 shutdown_connections 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.584+0000 7f6429342700 1 -- 172.21.15.12:0/4038112488 wait complete. 2023-09-29T23:18:50.586 INFO:teuthology.orchestra.run.smithi012.stderr:instructed osd(s) 5 to deep-scrub 2023-09-29T23:18:50.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:18:50.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.715+0000 7ff7b65f7700 10 monclient: get_monmap_and_config 2023-09-29T23:18:50.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.715+0000 7ff7b65f7700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.715 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.715+0000 7ff7b65f7700 10 monclient: monmap: 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:18:50.716329+0000 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:18:50.716329+0000 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.716 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:18:50.717 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 Processor -- start 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 -- start start 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient: init 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient: _add_conns ranks=[9,6,3,1,10,0,8,7,4,2,5] 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 0x7ff7b009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00a1f90 addr v2:172.21.15.12:3300/0 2023-09-29T23:18:50.726 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-d con 0x7ff7b00a2360 addr [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7b0133740 0x7ff7b0133be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-g con 0x7ff7b0133740 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 -- --> v2:172.21.15.12:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b009e350 con 0x7ff7b00a1f90 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b009ef70 con 0x7ff7b0133740 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 1 -- --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b009da70 con 0x7ff7b00a2360 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7b65f7700 10 monclient(hunting): authenticate will time out at 1440.076802s 2023-09-29T23:18:50.727 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 0x7ff7b009f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 1 --2- >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3303/0 says I am v2:172.21.15.12:37180/0 (socket says 172.21.15.12:37180) 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 1 -- 172.21.15.12:0/3650614135 learned_addr learned my addr 172.21.15.12:0/3650614135 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 1 --2- >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 0x7ff7b009f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51332/0 (socket says 172.21.15.12:51332) 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 10 monclient(hunting): get_auth_request con 0x7ff7b00a2360 auth_method 0 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7af7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 10 monclient(hunting): get_auth_request con 0x7ff7b00a1f90 auth_method 0 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.728 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.716+0000 7ff7affff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b4b94700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7b0133740 0x7ff7b0133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b4b94700 10 monclient(hunting): get_auth_request con 0x7ff7b0133740 auth_method 0 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b4b94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b4b94700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.729 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b4b94700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_done global_id 4956 payload 274 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 -- 172.21.15.12:0/3650614135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7b0133740 msgr2=0x7ff7b0133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7b0133740 0x7ff7b0133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 -- 172.21.15.12:0/3650614135 >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 msgr2=0x7ff7b009f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/3650614135 >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 0x7ff7b009f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 monclient: found mon.noname-d 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.12:3303/0 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 -- 172.21.15.12:0/3650614135 --> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0068ed0 con 0x7ff7b00a2360 2023-09-29T23:18:50.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009a10 tx=0x7ff7a4009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1f737bb0afadb614 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 1 -- 172.21.15.12:0/3650614135 <== mon.6 v2:172.21.15.12:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a400bc60 con 0x7ff7b00a2360 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: dump: 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.731 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b65f7700 5 monclient: authenticate success, global_id 4956 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b65f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 1 -- 172.21.15.12:0/3650614135 <== mon.6 v2:172.21.15.12:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a400bde0 con 0x7ff7b00a2360 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b65f7700 10 monclient: get_monmap_and_config success 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b65f7700 4 set_mon_vals no callback set 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 1 -- 172.21.15.12:0/3650614135 <== mon.6 v2:172.21.15.12:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a40118a0 con 0x7ff7b00a2360 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7aeffd700 10 monclient: dump: 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.733 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.734 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.717+0000 7ff7b65f7700 10 monclient: shutdown 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 -- 172.21.15.12:0/3650614135 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 msgr2=0x7ff7b00a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009a10 tx=0x7ff7a4009250 comp rx=0 tx=0).stop 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 -- 172.21.15.12:0/3650614135 shutdown_connections 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7b0133740 0x7ff7b0133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/3650614135 >> v2:172.21.15.12:3300/0 conn(0x7ff7b00a1f90 0x7ff7b009f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/3650614135 >> [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] conn(0x7ff7b00a2360 0x7ff7b00a27b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 -- 172.21.15.12:0/3650614135 >> 172.21.15.12:0/3650614135 conn(0x7ff7b00a3b50 msgr2=0x7ff7b00a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 -- 172.21.15.12:0/3650614135 shutdown_connections 2023-09-29T23:18:50.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 -- 172.21.15.12:0/3650614135 wait complete. 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 10 monclient: build_initial_monmap 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 10 monclient: monmap: 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:18:50.736 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.718+0000 7ff7b65f7700 1 Processor -- start 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 -- start start 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient: init 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient: _reopen_session rank -1 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient: _add_conns ranks=[0,8,4,3,6,2,1,7,5] 2023-09-29T23:18:50.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00a1f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-e con 0x7ff7b00a2360 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 0x7ff7b012d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): picked mon.noname-c con 0x7ff7b0133740 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0130ff0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b009e550 con 0x7ff7b0133740 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): start opening mon connection 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b012f800 con 0x7ff7b00a2360 2023-09-29T23:18:50.738 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): _renew_subs 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b65f7700 10 monclient(hunting): authenticate will time out at 1440.079532s 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51342/0 (socket says 172.21.15.12:51342) 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 0x7ff7b012d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:55332/0 (socket says 172.21.15.12:55332) 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 -- 172.21.15.12:0/2069023331 learned_addr learned my addr 172.21.15.12:0/2069023331 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): get_auth_request con 0x7ff7b00a1f90 auth_method 0 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): get_auth_request con 0x7ff7b00a2360 auth_method 0 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): get_auth_request con 0x7ff7b0133740 auth_method 0 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7b4b94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient(hunting): handle_auth_done global_id 5220 payload 824 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 -- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 msgr2=0x7ff7b0132ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 -- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 msgr2=0x7ff7b012d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 0x7ff7b012d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient: _finish_hunting 0 2023-09-29T23:18:50.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 monclient: found mon.noname-a 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.719+0000 7ff7affff700 1 -- 172.21.15.12:0/2069023331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40094f0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b4b94700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 0x7ff7b012d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7affff700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000e930 tx=0x7ff7a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a0013b00 con 0x7ff7b00a1f90 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: dump: 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.742 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.743 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: _finish_auth 0 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a0013ca0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 5 monclient: authenticate success, global_id 5220 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 10 monclient: _renew_subs 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b004c960 con 0x7ff7b00a1f90 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a0013e10 con 0x7ff7b00a1f90 2023-09-29T23:18:50.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7ae7fc700 4 set_mon_vals no callback set 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff797fff700 10 monclient: dump: 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:18:50.745 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 10 monclient: _renew_subs 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.720+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01432a0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff7a0013680 con 0x7ff7b00a1f90 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7b65f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7b65f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7b65f7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff797fff700 1 --2- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 0x7ff79c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b0070bc0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff7a00476e0 con 0x7ff7b00a1f90 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 0x7ff79c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.721+0000 7ff7af7fe700 10 monclient: get_auth_request con 0x7ff79c036850 auth_method 0 2023-09-29T23:18:50.747 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.722+0000 7ff7af7fe700 1 --2- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 0x7ff79c038d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009fd0 tx=0x7ff7a4017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:18:50.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.726+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7a0013930 con 0x7ff7b00a1f90 2023-09-29T23:18:50.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.726+0000 7ff797fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:18:50.748 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.726+0000 7ff797fff700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:18:50.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.914+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7b0069190 con 0x7ff79c036850 2023-09-29T23:18:50.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.917+0000 7ff797fff700 1 -- 172.21.15.12:0/2069023331 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52063 (secure 0 0 0) 0x7ff7a4006c50 con 0x7ff79c036850 2023-09-29T23:18:50.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:18:50.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 msgr2=0x7ff79c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 0x7ff79c038d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009fd0 tx=0x7ff7a4017000 comp rx=0 tx=0).stop 2023-09-29T23:18:50.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 10 monclient: shutdown 2023-09-29T23:18:50.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:18:50.919 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 msgr2=0x7ff7b0142420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000e930 tx=0x7ff7a00091a0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 shutdown_connections 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> v2:172.21.15.153:6812/83257 conn(0x7ff79c036850 0x7ff79c038d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7ff7b0133740 0x7ff7b012d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff7b00a1f90 0x7ff7b0142420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 --2- 172.21.15.12:0/2069023331 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7ff7b00a2360 0x7ff7b0132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 >> 172.21.15.12:0/2069023331 conn(0x7ff7b00a3b50 msgr2=0x7ff7b0135de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 shutdown_connections 2023-09-29T23:18:50.920 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:18:50.919+0000 7ff7b65f7700 1 -- 172.21.15.12:0/2069023331 wait complete. 2023-09-29T23:18:50.921 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:18:50.930 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2023-09-29T23:18:50.792572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4776460,"kb_used_data":557419820,"kb_used_omap":412,"kb_used_meta":0,"kb_avail":557419820,"statfs":{"total":575688990720,"available":570797895680,"internally_reserved":0,"allocated":570797895680,"data_stored":570797895680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":422822,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:13:39.134780+0000","last_change":"2023-09-29T23:13:38.129087+0000","last_active":"2023-09-29T23:13:39.134780+0000","last_peered":"2023-09-29T23:13:39.134780+0000","last_clean":"2023-09-29T23:13:39.134780+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:13:39.134780+0000","last_undegraded":"2023-09-29T23:13:39.134780+0000","last_fullsized":"2023-09-29T23:13:39.134780+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126311+0000","last_change":"2023-09-29T23:13:38.126049+0000","last_active":"2023-09-29T23:13:38.126311+0000","last_peered":"2023-09-29T23:13:38.126311+0000","last_clean":"2023-09-29T23:13:38.126311+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:13:38.126311+0000","last_undegraded":"2023-09-29T23:13:38.126311+0000","last_fullsized":"2023-09-29T23:13:38.126311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:20.078748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:13:39.135872+0000","last_change":"2023-09-29T23:13:38.130385+0000","last_active":"2023-09-29T23:13:39.135872+0000","last_peered":"2023-09-29T23:13:39.135872+0000","last_clean":"2023-09-29T23:13:39.135872+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:13:39.135872+0000","last_undegraded":"2023-09-29T23:13:39.135872+0000","last_fullsized":"2023-09-29T23:13:39.135872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.125665+0000","last_change":"2023-09-29T23:13:38.125481+0000","last_active":"2023-09-29T23:13:38.125665+0000","last_peered":"2023-09-29T23:13:38.125665+0000","last_clean":"2023-09-29T23:13:38.125665+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:13:38.125665+0000","last_undegraded":"2023-09-29T23:13:38.125665+0000","last_fullsized":"2023-09-29T23:13:38.125665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:40:37.808431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:13:39.134708+0000","last_change":"2023-09-29T23:13:34.744453+0000","last_active":"2023-09-29T23:13:39.134708+0000","last_peered":"2023-09-29T23:13:39.134708+0000","last_clean":"2023-09-29T23:13:39.134708+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:13:39.134708+0000","last_undegraded":"2023-09-29T23:13:39.134708+0000","last_fullsized":"2023-09-29T23:13:39.134708+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:33.720374+0000","last_clean_scrub_stamp":"2023-09-29T23:13:33.720374+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:13:39.134827+0000","last_change":"2023-09-29T23:13:38.128543+0000","last_active":"2023-09-29T23:13:39.134827+0000","last_peered":"2023-09-29T23:13:39.134827+0000","last_clean":"2023-09-29T23:13:39.134827+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:13:39.134827+0000","last_undegraded":"2023-09-29T23:13:39.134827+0000","last_fullsized":"2023-09-29T23:13:39.134827+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.126178+0000","last_change":"2023-09-29T23:13:38.125121+0000","last_active":"2023-09-29T23:13:38.126178+0000","last_peered":"2023-09-29T23:13:38.126178+0000","last_clean":"2023-09-29T23:13:38.126178+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:13:38.126178+0000","last_undegraded":"2023-09-29T23:13:38.126178+0000","last_fullsized":"2023-09-29T23:13:38.126178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:35.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:13:39.134164+0000","last_change":"2023-09-29T23:13:38.125939+0000","last_active":"2023-09-29T23:13:39.134164+0000","last_peered":"2023-09-29T23:13:39.134164+0000","last_clean":"2023-09-29T23:13:39.134164+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:13:39.134164+0000","last_undegraded":"2023-09-29T23:13:39.134164+0000","last_fullsized":"2023-09-29T23:13:39.134164+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-29T23:13:38.127116+0000","last_change":"2023-09-29T23:13:38.126972+0000","last_active":"2023-09-29T23:13:38.127116+0000","last_peered":"2023-09-29T23:13:38.127116+0000","last_clean":"2023-09-29T23:13:38.127116+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:13:38.127116+0000","last_undegraded":"2023-09-29T23:13:38.127116+0000","last_fullsized":"2023-09-29T23:13:38.127116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:13:37.110438+0000","last_clean_scrub_stamp":"2023-09-29T23:13:37.110438+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:29:50.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796360,"kb_used_data":92903020,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92903020,"statfs":{"total":95948165120,"available":95132692480,"internally_reserved":0,"allocated":95132692480,"data_stored":95132692480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119092,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.38900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.505,"15min":0.50800000000000001},"min":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.60199999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.67100000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67900000000000005,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.30099999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59699999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.376},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.498,"5min":0.627,"15min":0.627},"last":0.311}]},{"osd":4,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.40300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57699999999999996,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.40999999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640322,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.378},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.50800000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60799999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.443},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.48499999999999999}]},{"osd":3,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.31}]},{"osd":5,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59599999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49199999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902384,"kb_used_omap":115,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":0,"allocated":95132041216,"data_stored":95132041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118413,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34399999999999997},"min":{"1min":0.23300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53800000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.57499999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.504,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.45100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67700000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.67200000000000004}]},{"osd":4,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.94599999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.627},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.46999999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.57199999999999995},"min":{"1min":0.50900000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.41799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.71599999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43684,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.432},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.34499999999999997}]},{"osd":1,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68300000000000005,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.754,"15min":0.754},"last":0.44600000000000001}]},{"osd":2,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63300000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.63400000000000001},"min":{"1min":0.38500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61599999999999999,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40600000000000003,"15min":0.41699999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.51200000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.46700000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60899999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.622,"5min":0.63,"15min":0.63},"last":0.376},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58999999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.30499999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48965,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46200000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.48699999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.254}]},{"osd":2,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.35499999999999998},"min":{"1min":0.26300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.44400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.316,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.32900000000000001}]},{"osd":3,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78400000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.44600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.47899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79100000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57299999999999995,"15min":0.60999999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60499999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43703,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.54900000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":1.115,"15min":1.115},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53300000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36499999999999999},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.29099999999999998}]},{"osd":3,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.74299999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.498},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.753,"5min":1.601,"15min":1.601},"last":0.52600000000000002}]},{"osd":5,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80300000000000005,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91500000000000004,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.56499999999999995}]}]}],"pool_statfs":[]}} 2023-09-29T23:18:50.934 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.934 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-29T23:13:33.720374+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=33, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-29T23:13:37.110438+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=272, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=29, tm_hour=23, tm_min=18, tm_sec=46, tm_wday=4, tm_yday=272, tm_isdst=0) 2023-09-29T23:18:50.935 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-29T23:19:10.937 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 10 monclient: get_monmap_and_config 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 10 monclient: build_initial_monmap 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 10 monclient: monmap: 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:19:11.066954+0000 2023-09-29T23:19:11.066 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:19:11.066954+0000 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:19:11.067 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:19:11.068 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:19:11.068 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 1 Processor -- start 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 1 -- start start 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.065+0000 7f4511b5d700 10 monclient: init 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient: _reopen_session rank -1 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient: _add_conns ranks=[3,0,8,4,5,1,7,9,2,10,6] 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c1303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-g con 0x7f450c06ed80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f450c06f150 addr v1:172.21.15.12:6789/0 2023-09-29T23:19:11.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c09e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-e con 0x7f450c06f540 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 1 -- --> v1:172.21.15.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c068ed0 con 0x7f450c06f150 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c09fc20 con 0x7f450c06ed80 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f450c06d040 con 0x7f450c06f540 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): _renew_subs 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f4511b5d700 10 monclient(hunting): authenticate will time out at 1460.427383s 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c09e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450affd700 1 --1- >> v1:172.21.15.12:6789/0 conn(0x7f450c06f150 0x7f450c09dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.12:6789/0 says I am v1:172.21.15.12:57038/0 (socket says 172.21.15.12:57038) 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450affd700 1 -- 172.21.15.12:0/3082636870 learned_addr learned my addr 172.21.15.12:0/3082636870 (peer_addr_for_me v1:172.21.15.12:0/0) 2023-09-29T23:19:11.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): get_auth_request con 0x7f450c06f540 auth_method 0 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c1303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): get_auth_request con 0x7f450c06ed80 auth_method 0 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.078 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450a7fc700 1 -- 172.21.15.12:0/3082636870 <== mon.0 v1:172.21.15.12:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389154839 0 0) 0x7f450c068ed0 con 0x7f450c06f150 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450a7fc700 10 monclient(hunting): my global_id is 5229 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450a7fc700 1 -- 172.21.15.12:0/3082636870 --> v1:172.21.15.12:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8003570 con 0x7f450c06f150 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient(hunting): handle_auth_done global_id 4877 payload 274 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 -- 172.21.15.12:0/3082636870 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 msgr2=0x7f450c1303e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:19:11.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c1303e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 -- 172.21.15.12:0/3082636870 >> v1:172.21.15.12:6789/0 conn(0x7f450c06f150 legacy=0x7f450c09dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 monclient: found mon.noname-e 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.066+0000 7f450bfff700 1 -- 172.21.15.12:0/3082636870 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c04c960 con 0x7f450c06f540 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450bfff700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c09e430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4504009ec0 tx=0x7f4504009780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b13ccd08d881a298 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 1 -- 172.21.15.12:0/3082636870 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f450400e070 con 0x7f450c06f540 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:19:11.080 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: dump: 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.081 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: _finish_auth 0 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 1 -- 172.21.15.12:0/3082636870 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504012430 con 0x7f450c06f540 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 5 monclient: authenticate success, global_id 4877 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:19:11.082 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 10 monclient: get_monmap_and_config success 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 1 -- 172.21.15.12:0/3082636870 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4504012a20 con 0x7f450c06f540 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 4 set_mon_vals no callback set 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f450a7fc700 10 monclient: dump: 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.083 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 10 monclient: shutdown 2023-09-29T23:19:11.084 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 -- 172.21.15.12:0/3082636870 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 msgr2=0x7f450c09e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c09e430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4504009ec0 tx=0x7f4504009780 comp rx=0 tx=0).stop 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 -- 172.21.15.12:0/3082636870 shutdown_connections 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c09e430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 --2- 172.21.15.12:0/3082636870 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c1303e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 -- 172.21.15.12:0/3082636870 >> 172.21.15.12:0/3082636870 conn(0x7f450c06aab0 msgr2=0x7f450c06aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 -- 172.21.15.12:0/3082636870 shutdown_connections 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.067+0000 7f4511b5d700 1 -- 172.21.15.12:0/3082636870 wait complete. 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.068+0000 7f4511b5d700 10 monclient: build_initial_monmap 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.068+0000 7f4511b5d700 10 monclient: monmap: 2023-09-29T23:19:11.085 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:19:11.086 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.068+0000 7f4511b5d700 1 Processor -- start 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.068+0000 7f4511b5d700 1 -- start start 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.068+0000 7f4511b5d700 10 monclient: init 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient: _reopen_session rank -1 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient: _add_conns ranks=[3,4,8,1,7,6,5,0,2] 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c12bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-g con 0x7f450c06ed80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:19:11.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f450c06f150 0x7f450c12bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-c con 0x7f450c06f150 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): picked mon.noname-e con 0x7f450c06f540 addr [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c06cc50 con 0x7f450c06ed80 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f450c12d7e0 con 0x7f450c06f150 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 1 -- --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f450c12e3e0 con 0x7f450c06f540 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): _renew_subs 2023-09-29T23:19:11.088 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f4511b5d700 10 monclient(hunting): authenticate will time out at 1460.430408s 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 --2- >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3304/0 says I am v2:172.21.15.12:57098/0 (socket says 172.21.15.12:57098) 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 -- 172.21.15.12:0/2838535695 learned_addr learned my addr 172.21.15.12:0/2838535695 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): get_auth_request con 0x7f450c06f540 auth_method 0 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f450c06f150 0x7f450c12bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): get_auth_request con 0x7f450c06f150 auth_method 0 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450b7fe700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c12bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450b7fe700 10 monclient(hunting): get_auth_request con 0x7f450c06ed80 auth_method 0 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient(hunting): handle_auth_done global_id 4886 payload 824 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 -- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f450c06f150 msgr2=0x7f450c12bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f450c06f150 0x7f450c12bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 -- 172.21.15.12:0/2838535695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 msgr2=0x7f450c12bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c12bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient: _finish_hunting 0 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 monclient: found mon.noname-e 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:19:11.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.12:3304/0 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.069+0000 7f450bfff700 1 -- 172.21.15.12:0/2838535695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504009a20 con 0x7f450c06f540 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f450bfff700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4504013000 tx=0x7f4504019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4504003310 con 0x7f450c06f540 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: dump: 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.092 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: _finish_auth 0 2023-09-29T23:19:11.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 5 monclient: authenticate success, global_id 4886 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450400bec0 con 0x7f450c06f540 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 10 monclient: _renew_subs 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c119ac0 con 0x7f450c06f540 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4504012d50 con 0x7f450c06f540 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-29T23:19:11.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4509ffb700 4 set_mon_vals no callback set 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f45037fe700 10 monclient: dump: 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.095 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 10 monclient: _renew_subs 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.070+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c12a8a0 con 0x7f450c06f540 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f450402f450 con 0x7f450c06f540 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f4511b5d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f4511b5d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.096 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f4511b5d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.12:3304/0 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 --> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c12aaf0 con 0x7f450c06f540 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f45037fe700 1 --2- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 0x7f44ec038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4504048aa0 con 0x7f450c06f540 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f450b7fe700 1 --2- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 0x7f44ec038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.071+0000 7f450b7fe700 10 monclient: get_auth_request con 0x7f44ec036850 auth_method 0 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.072+0000 7f450b7fe700 1 --2- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 0x7f44ec038d00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009d40 tx=0x7f44f4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.076+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mon.8 v2:172.21.15.12:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f450c12aaf0 con 0x7f450c06f540 2023-09-29T23:19:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.076+0000 7f45037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.098 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.076+0000 7f45037fe700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:19:11.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.266+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 --> v2:172.21.15.153:6812/83257 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f450c069190 con 0x7f44ec036850 2023-09-29T23:19:11.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.268+0000 7f45037fe700 1 -- 172.21.15.12:0/2838535695 <== mgr.4111 v2:172.21.15.153:6812/83257 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52444 (secure 0 0 0) 0x7f44f4009e80 con 0x7f44ec036850 2023-09-29T23:19:11.270 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-09-29T23:19:11.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 msgr2=0x7f44ec038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 0x7f44ec038d00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009d40 tx=0x7f44f4008040 comp rx=0 tx=0).stop 2023-09-29T23:19:11.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 10 monclient: shutdown 2023-09-29T23:19:11.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:19:11.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 msgr2=0x7f450c12c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4504013000 tx=0x7f4504019000 comp rx=0 tx=0).stop 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 shutdown_connections 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] conn(0x7f450c06f540 0x7f450c12c530 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> v2:172.21.15.153:6812/83257 conn(0x7f44ec036850 0x7f44ec038d00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450c06ed80 0x7f450c12bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 --2- 172.21.15.12:0/2838535695 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7f450c06f150 0x7f450c12bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 >> 172.21.15.12:0/2838535695 conn(0x7f450c06aab0 msgr2=0x7f450c0a1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 shutdown_connections 2023-09-29T23:19:11.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.270+0000 7f4511b5d700 1 -- 172.21.15.12:0/2838535695 wait complete. 2023-09-29T23:19:11.274 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-09-29T23:19:11.283 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2023-09-29T23:19:10.796785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":4775824,"kb_used_data":557420456,"kb_used_omap":438,"kb_used_meta":0,"kb_avail":557420456,"statfs":{"total":575688990720,"available":570798546944,"internally_reserved":0,"allocated":570798546944,"data_stored":570798546944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":448737,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:19:01.652058+0000","last_change":"2023-09-29T23:19:01.652019+0000","last_active":"2023-09-29T23:19:01.652058+0000","last_peered":"2023-09-29T23:19:01.652058+0000","last_clean":"2023-09-29T23:19:01.652058+0000","last_became_active":"2023-09-29T23:13:38.128798+0000","last_became_peered":"2023-09-29T23:13:38.128798+0000","last_unstale":"2023-09-29T23:19:01.652058+0000","last_undegraded":"2023-09-29T23:19:01.652058+0000","last_fullsized":"2023-09-29T23:19:01.652058+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:19:01.651877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:19:01.651877+0000","last_clean_scrub_stamp":"2023-09-29T23:19:01.651877+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:14:41.251673+0000","scrub_duration":0.00158541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:51.239998+0000","last_change":"2023-09-29T23:18:51.239976+0000","last_active":"2023-09-29T23:18:51.239998+0000","last_peered":"2023-09-29T23:18:51.239998+0000","last_clean":"2023-09-29T23:18:51.239998+0000","last_became_active":"2023-09-29T23:13:38.125783+0000","last_became_peered":"2023-09-29T23:13:38.125783+0000","last_unstale":"2023-09-29T23:18:51.239998+0000","last_undegraded":"2023-09-29T23:18:51.239998+0000","last_fullsized":"2023-09-29T23:18:51.239998+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:51.239905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:51.239905+0000","last_clean_scrub_stamp":"2023-09-29T23:18:51.239905+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:22:41.009337+0000","scrub_duration":0.0017806059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:54.680914+0000","last_change":"2023-09-29T23:18:54.680880+0000","last_active":"2023-09-29T23:18:54.680914+0000","last_peered":"2023-09-29T23:18:54.680914+0000","last_clean":"2023-09-29T23:18:54.680914+0000","last_became_active":"2023-09-29T23:13:38.130122+0000","last_became_peered":"2023-09-29T23:13:38.130122+0000","last_unstale":"2023-09-29T23:18:54.680914+0000","last_undegraded":"2023-09-29T23:18:54.680914+0000","last_fullsized":"2023-09-29T23:18:54.680914+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:54.680797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:54.680797+0000","last_clean_scrub_stamp":"2023-09-29T23:18:54.680797+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:18:35.238493+0000","scrub_duration":0.0022234189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:53.913530+0000","last_change":"2023-09-29T23:18:53.913492+0000","last_active":"2023-09-29T23:18:53.913530+0000","last_peered":"2023-09-29T23:18:53.913530+0000","last_clean":"2023-09-29T23:18:53.913530+0000","last_became_active":"2023-09-29T23:13:38.125243+0000","last_became_peered":"2023-09-29T23:13:38.125243+0000","last_unstale":"2023-09-29T23:18:53.913530+0000","last_undegraded":"2023-09-29T23:18:53.913530+0000","last_fullsized":"2023-09-29T23:18:53.913530+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:53.913379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:53.913379+0000","last_clean_scrub_stamp":"2023-09-29T23:18:53.913379+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:57.440563+0000","scrub_duration":0.0020228830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'119","reported_seq":175,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:51.692259+0000","last_change":"2023-09-29T23:18:51.692230+0000","last_active":"2023-09-29T23:18:51.692259+0000","last_peered":"2023-09-29T23:18:51.692259+0000","last_clean":"2023-09-29T23:18:51.692259+0000","last_became_active":"2023-09-29T23:13:34.744215+0000","last_became_peered":"2023-09-29T23:13:34.744215+0000","last_unstale":"2023-09-29T23:18:51.692259+0000","last_undegraded":"2023-09-29T23:18:51.692259+0000","last_fullsized":"2023-09-29T23:18:51.692259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'119","last_scrub_stamp":"2023-09-29T23:18:51.692163+0000","last_deep_scrub":"18'119","last_deep_scrub_stamp":"2023-09-29T23:18:51.692163+0000","last_clean_scrub_stamp":"2023-09-29T23:18:51.692163+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:28:01.334847+0000","scrub_duration":0.0053869859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:59.623352+0000","last_change":"2023-09-29T23:18:59.623308+0000","last_active":"2023-09-29T23:18:59.623352+0000","last_peered":"2023-09-29T23:18:59.623352+0000","last_clean":"2023-09-29T23:18:59.623352+0000","last_became_active":"2023-09-29T23:13:38.128307+0000","last_became_peered":"2023-09-29T23:13:38.128307+0000","last_unstale":"2023-09-29T23:18:59.623352+0000","last_undegraded":"2023-09-29T23:18:59.623352+0000","last_fullsized":"2023-09-29T23:18:59.623352+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:59.623227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:59.623227+0000","last_clean_scrub_stamp":"2023-09-29T23:18:59.623227+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:09:19.924779+0000","scrub_duration":0.0015700860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:49.900793+0000","last_change":"2023-09-29T23:18:49.900768+0000","last_active":"2023-09-29T23:18:49.900793+0000","last_peered":"2023-09-29T23:18:49.900793+0000","last_clean":"2023-09-29T23:18:49.900793+0000","last_became_active":"2023-09-29T23:13:38.124842+0000","last_became_peered":"2023-09-29T23:13:38.124842+0000","last_unstale":"2023-09-29T23:18:49.900793+0000","last_undegraded":"2023-09-29T23:18:49.900793+0000","last_fullsized":"2023-09-29T23:18:49.900793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:49.900675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:49.900675+0000","last_clean_scrub_stamp":"2023-09-29T23:18:49.900675+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:12:18.439371+0000","scrub_duration":0.0014859719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:54.509693+0000","last_change":"2023-09-29T23:18:54.509671+0000","last_active":"2023-09-29T23:18:54.509693+0000","last_peered":"2023-09-29T23:18:54.509693+0000","last_clean":"2023-09-29T23:18:54.509693+0000","last_became_active":"2023-09-29T23:13:38.125680+0000","last_became_peered":"2023-09-29T23:13:38.125680+0000","last_unstale":"2023-09-29T23:18:54.509693+0000","last_undegraded":"2023-09-29T23:18:54.509693+0000","last_fullsized":"2023-09-29T23:18:54.509693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:54.509609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:54.509609+0000","last_clean_scrub_stamp":"2023-09-29T23:18:54.509609+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:33.408281+0000","scrub_duration":0.001807482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:18:50.625507+0000","last_change":"2023-09-29T23:18:50.625449+0000","last_active":"2023-09-29T23:18:50.625507+0000","last_peered":"2023-09-29T23:18:50.625507+0000","last_clean":"2023-09-29T23:18:50.625507+0000","last_became_active":"2023-09-29T23:13:38.126702+0000","last_became_peered":"2023-09-29T23:13:38.126702+0000","last_unstale":"2023-09-29T23:18:50.625507+0000","last_undegraded":"2023-09-29T23:18:50.625507+0000","last_fullsized":"2023-09-29T23:18:50.625507+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:18:50.625345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:18:50.625345+0000","last_clean_scrub_stamp":"2023-09-29T23:18:50.625345+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:01:48.075151+0000","scrub_duration":0.0017377969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796360,"kb_used_data":92903020,"kb_used_omap":122,"kb_used_meta":0,"kb_avail":92903020,"statfs":{"total":95948165120,"available":95132692480,"internally_reserved":0,"allocated":95132692480,"data_stored":95132692480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125920,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.504}]},{"osd":1,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.38900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.505,"15min":0.50800000000000001},"min":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.60199999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.67100000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67900000000000005,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.30099999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59699999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.498,"5min":0.627,"15min":0.627},"last":0.36099999999999999}]},{"osd":4,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.40300000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.38},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57699999999999996,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.48599999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640326,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52688,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69099999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.50800000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60799999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61099999999999999}]},{"osd":1,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.312,"15min":0.312},"max":{"1min":2.129,"5min":2.129,"15min":2.129},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.441},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Sep 29 23:18:41 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.629,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.44500000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.59599999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.501}]}]},{"osd":1,"up_from":11,"seq":47244640325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796360,"kb_used_data":92903020,"kb_used_omap":122,"kb_used_meta":0,"kb_avail":92903020,"statfs":{"total":95948165120,"available":95132692480,"internally_reserved":0,"allocated":95132692480,"data_stored":95132692480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125072,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34399999999999997},"min":{"1min":0.23300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53800000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.57499999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.37}]},{"osd":2,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.504,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.45100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67700000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.94599999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.46999999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Sep 29 23:18:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.57199999999999995},"min":{"1min":0.50900000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.41799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55600000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":45,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46125,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52600000000000002,"15min":0.53600000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68300000000000005,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.754,"15min":0.754},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.52400000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63300000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.63400000000000001},"min":{"1min":0.38500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.61599999999999999,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.498}]},{"osd":4,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40600000000000003,"15min":0.41699999999999998},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.51200000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.46700000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60899999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41899999999999998}]},{"osd":5,"last update":"Fri Sep 29 23:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.622,"5min":0.63,"15min":0.63},"last":0.374},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58999999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.38800000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52738,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.46200000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.48699999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.39100000000000001}]},{"osd":2,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.35499999999999998},"min":{"1min":0.26300000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.44400000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.316,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.50900000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78400000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55300000000000005,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.44600000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.47899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79100000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.49199999999999999}]},{"osd":5,"last update":"Fri Sep 29 23:18:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57299999999999995,"15min":0.60999999999999999},"min":{"1min":0.46200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51400000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903604,"kb_used_omap":45,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":0,"allocated":95133290496,"data_stored":95133290496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46194,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.54900000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52400000000000002,"5min":1.115,"15min":1.115},"last":0.27100000000000002}]},{"osd":1,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53300000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36199999999999999,"15min":0.36499999999999999},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.28699999999999998}]},{"osd":3,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.74299999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.623},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57899999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.58299999999999996},"min":{"1min":0.44800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.753,"5min":1.601,"15min":1.601},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Sep 29 23:18:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80300000000000005,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91500000000000004,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.60599999999999998}]}]}],"pool_statfs":[]}} 2023-09-29T23:19:11.285 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-29T23:19:11.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.402+0000 7fdc84a1e700 10 monclient: get_monmap_and_config 2023-09-29T23:19:11.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.402+0000 7fdc84a1e700 10 monclient: build_initial_monmap 2023-09-29T23:19:11.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.402+0000 7fdc84a1e700 10 monclient: monmap: 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:19:11.404416+0000 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:19:11.404416+0000 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:0: v1:172.21.15.12:6789/0 mon.noname-a-legacy 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f-legacy 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:19:11.404 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:9: v2:172.21.15.12:3300/0 mon.noname-a 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr:10: v2:172.21.15.153:3300/0 mon.noname-f 2023-09-29T23:19:11.405 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 Processor -- start 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 -- start start 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient: init 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient: _reopen_session rank -1 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient: _add_conns ranks=[5,4,1,3,0,2,9,7,6,10,8] 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 --2- >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdc800a0200 0x7fdc800a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-h con 0x7fdc800a0200 addr [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] 2023-09-29T23:19:11.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-c con 0x7fdc8006f410 addr [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdc800a05d0 addr v1:172.21.15.153:6789/0 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc80068ed0 con 0x7fdc800a05d0 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 -- --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8009cab0 con 0x7fdc8006f410 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 1 -- --> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8009c4f0 con 0x7fdc800a0200 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): _renew_subs 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc84a1e700 10 monclient(hunting): authenticate will time out at 1460.764677s 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 1 --2- >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3302/0 says I am v2:172.21.15.12:38846/0 (socket says 172.21.15.12:38846) 2023-09-29T23:19:11.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 1 -- 172.21.15.12:0/1889974829 learned_addr learned my addr 172.21.15.12:0/1889974829 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): get_auth_request con 0x7fdc8006f410 auth_method 0 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.403+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7ed9d700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdc800a0200 0x7fdc800a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7ed9d700 10 monclient(hunting): get_auth_request con 0x7fdc800a0200 auth_method 0 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_done global_id 5044 payload 274 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 -- 172.21.15.12:0/1889974829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdc800a0200 msgr2=0x7fdc800a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdc800a0200 0x7fdc800a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 -- 172.21.15.12:0/1889974829 >> v1:172.21.15.153:6789/0 conn(0x7fdc800a05d0 legacy=0x7fdc80139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 10 monclient: _finish_hunting 0 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 monclient: found mon.noname-c 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3302/0 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 -- 172.21.15.12:0/1889974829 --> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8004c960 con 0x7fdc8006f410 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7e59c700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc740099e0 tx=0x7fdc74009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4b45a1df63f77cea server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 1 -- 172.21.15.12:0/1889974829 <== mon.4 v2:172.21.15.12:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdc7400bce0 con 0x7fdc8006f410 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: dump: 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.418 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: _finish_auth 0 2023-09-29T23:19:11.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:19:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 5 monclient: authenticate success, global_id 5044 2023-09-29T23:19:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 1 -- 172.21.15.12:0/1889974829 <== mon.4 v2:172.21.15.12:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400be60 con 0x7fdc8006f410 2023-09-29T23:19:11.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 10 monclient: get_monmap_and_config success 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 1 -- 172.21.15.12:0/1889974829 <== mon.4 v2:172.21.15.12:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdc740118a0 con 0x7fdc8006f410 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 4 set_mon_vals no callback set 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc7dd9b700 10 monclient: dump: 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.422 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 10 monclient: shutdown 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 1 -- 172.21.15.12:0/1889974829 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 msgr2=0x7fdc800a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.404+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc740099e0 tx=0x7fdc74009250 comp rx=0 tx=0).stop 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 -- 172.21.15.12:0/1889974829 shutdown_connections 2023-09-29T23:19:11.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] conn(0x7fdc800a0200 0x7fdc800a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/1889974829 >> [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] conn(0x7fdc8006f410 0x7fdc800a1350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 -- 172.21.15.12:0/1889974829 >> 172.21.15.12:0/1889974829 conn(0x7fdc80063bf0 msgr2=0x7fdc800a6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 -- 172.21.15.12:0/1889974829 shutdown_connections 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 1 -- 172.21.15.12:0/1889974829 wait complete. 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 10 monclient: build_initial_monmap 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.405+0000 7fdc84a1e700 10 monclient: monmap: 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 0 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 0.000000 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:created 0.000000 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 0 (unknown) 2023-09-29T23:19:11.424 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 1 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-f 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-b 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-g 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.noname-c 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.noname-h 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.noname-d 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.noname-i 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.noname-e 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 Processor -- start 2023-09-29T23:19:11.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 -- start start 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient: init 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient: _reopen_session rank -1 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient: _add_conns ranks=[1,7,3,2,6,5,0,4,8] 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 0x7fdc80132b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-f con 0x7fdc800a0200 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-i con 0x7fdc800a05d0 addr [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc80133070 0x7fdc8011cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): picked mon.noname-g con 0x7fdc80133070 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8011fc50 con 0x7fdc800a0200 2023-09-29T23:19:11.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80142fc0 con 0x7fdc80133070 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): start opening mon connection 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 1 -- --> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8009b8b0 con 0x7fdc800a05d0 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): _renew_subs 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc84a1e700 10 monclient(hunting): authenticate will time out at 1460.767536s 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 0x7fdc80132b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 1 --2- >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3303/0 says I am v2:172.21.15.12:45782/0 (socket says 172.21.15.12:45782) 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 1 -- 172.21.15.12:0/2637675784 learned_addr learned my addr 172.21.15.12:0/2637675784 (peer_addr_for_me v2:172.21.15.12:0/0) 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): get_auth_request con 0x7fdc800a05d0 auth_method 0 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): get_auth_request con 0x7fdc800a0200 auth_method 0 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc80133070 0x7fdc8011cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): get_auth_request con 0x7fdc80133070 auth_method 0 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): _init_auth method 2 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.406+0000 7fdc7f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 10 monclient(hunting): handle_auth_done global_id 5221 payload 824 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 -- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 msgr2=0x7fdc8011ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 -- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc80133070 msgr2=0x7fdc8011cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.429 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc80133070 0x7fdc8011cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 10 monclient: _finish_hunting 0 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 monclient: found mon.noname-f 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.153:3300/0 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 -- 172.21.15.12:0/2637675784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc740094f0 con 0x7fdc800a0200 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7e59c700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7ed9d700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 0x7fdc80132b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000d890 tx=0x7fdc70008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdc700073c0 con 0x7fdc800a0200 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-29T23:19:11.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: dump: 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.431 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: _finish_auth 0 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc70012a70 con 0x7fdc800a0200 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 5 monclient: authenticate success, global_id 5221 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 10 monclient: _renew_subs 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8006f7f0 con 0x7fdc800a0200 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-29T23:19:11.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdc70012be0 con 0x7fdc800a0200 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc7d59a700 4 set_mon_vals no callback set 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc66ffd700 10 monclient: dump: 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:epoch 1 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:fsid b8fec2b7-0020-4de5-aeb3-24af8bd53167 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:last_changed 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:created 2023-09-29T23:13:10.833638+0000 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:min_mon_release 17 (quincy) 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:election_strategy: 3 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.f 2023-09-29T23:19:11.433 INFO:teuthology.orchestra.run.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.b 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:3: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.g 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:4: [v2:172.21.15.12:3302/0,v1:172.21.15.12:6791/0] mon.c 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:5: [v2:172.21.15.153:3302/0,v1:172.21.15.153:6791/0] mon.h 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:6: [v2:172.21.15.12:3303/0,v1:172.21.15.12:6792/0] mon.d 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:7: [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] mon.i 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:8: [v2:172.21.15.12:3304/0,v1:172.21.15.12:6793/0] mon.e 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 10 monclient: _renew_subs 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.407+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8011ed60 con 0x7fdc800a0200 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc84a1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc84a1e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc84a1e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fdc70012d40 con 0x7fdc800a0200 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8011fe70 con 0x7fdc800a0200 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc66ffd700 1 --2- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 0x7fdc6c038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdc70047580 con 0x7fdc800a0200 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc7e59c700 1 --2- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 0x7fdc6c038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.408+0000 7fdc7e59c700 10 monclient: get_auth_request con 0x7fdc6c036850 auth_method 0 2023-09-29T23:19:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.409+0000 7fdc7e59c700 1 --2- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 0x7fdc6c038d00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009880 tx=0x7fdc74017000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:19:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.413+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc700125f0 con 0x7fdc800a0200 2023-09-29T23:19:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.413+0000 7fdc66ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-29T23:19:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.413+0000 7fdc66ffd700 10 monclient: _finish_command 1 = system:0 2023-09-29T23:19:11.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.606+0000 7fdc84a1e700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-29T23:19:11.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.606+0000 7fdc84a1e700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-29T23:19:11.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.606+0000 7fdc84a1e700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.153:3300/0 2023-09-29T23:19:11.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.606+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fdc80070bc0 con 0x7fdc800a0200 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fdc700125f0 con 0x7fdc800a0200 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc66ffd700 10 monclient: handle_config config(1 keys) v1 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc7d59a700 4 set_mon_vals no callback set 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc7d59a700 10 set_mon_vals mon_health_to_clog = false 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc66ffd700 1 -- 172.21.15.12:0/2637675784 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fdc70011e60 con 0x7fdc800a0200 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc66ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-29T23:19:11.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.613+0000 7fdc66ffd700 10 monclient: _finish_command 2 = system:0 2023-09-29T23:19:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 msgr2=0x7fdc6c038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 0x7fdc6c038d00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009880 tx=0x7fdc74017000 comp rx=0 tx=0).stop 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 10 monclient: shutdown 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 msgr2=0x7fdc80132b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 0x7fdc80132b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000d890 tx=0x7fdc70008040 comp rx=0 tx=0).stop 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 shutdown_connections 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> v2:172.21.15.153:6812/83257 conn(0x7fdc6c036850 0x7fdc6c038d00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc80133070 0x7fdc8011cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc800a0200 0x7fdc80132b30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 --2- 172.21.15.12:0/2637675784 >> [v2:172.21.15.153:3303/0,v1:172.21.15.153:6792/0] conn(0x7fdc800a05d0 0x7fdc8011ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:19:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 >> 172.21.15.12:0/2637675784 conn(0x7fdc80063bf0 msgr2=0x7fdc800a4870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:19:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 shutdown_connections 2023-09-29T23:19:11.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-09-29T23:19:11.615+0000 7fdc84a1e700 1 -- 172.21.15.12:0/2637675784 wait complete. 2023-09-29T23:19:11.628 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:19:11.629 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-29T23:19:11.629 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-29T23:19:11.629 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-29T23:19:11.629 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.639 INFO:tasks.ceph.osd.0:Stopped 2023-09-29T23:19:11.639 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-29T23:19:11.640 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.691 INFO:tasks.ceph.osd.1:Stopped 2023-09-29T23:19:11.691 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-29T23:19:11.691 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.741 INFO:tasks.ceph.osd.2:Stopped 2023-09-29T23:19:11.741 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-29T23:19:11.741 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.754 INFO:tasks.ceph.osd.3:Stopped 2023-09-29T23:19:11.754 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-29T23:19:11.755 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.807 INFO:tasks.ceph.osd.4:Stopped 2023-09-29T23:19:11.807 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-29T23:19:11.807 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.859 INFO:tasks.ceph.osd.5:Stopped 2023-09-29T23:19:11.859 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-29T23:19:11.859 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-29T23:19:11.859 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.925 INFO:tasks.ceph.mgr.x:Stopped 2023-09-29T23:19:11.925 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-29T23:19:11.926 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-29T23:19:11.926 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.936 INFO:tasks.ceph.mon.a:Stopped 2023-09-29T23:19:11.936 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-29T23:19:11.936 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:11.987 INFO:tasks.ceph.mon.b:Stopped 2023-09-29T23:19:11.987 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-29T23:19:11.987 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.038 INFO:tasks.ceph.mon.c:Stopped 2023-09-29T23:19:12.039 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2023-09-29T23:19:12.039 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.090 INFO:tasks.ceph.mon.d:Stopped 2023-09-29T23:19:12.090 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2023-09-29T23:19:12.090 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.140 INFO:tasks.ceph.mon.e:Stopped 2023-09-29T23:19:12.141 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2023-09-29T23:19:12.141 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.152 INFO:tasks.ceph.mon.f:Stopped 2023-09-29T23:19:12.153 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2023-09-29T23:19:12.153 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.205 INFO:tasks.ceph.mon.g:Stopped 2023-09-29T23:19:12.205 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2023-09-29T23:19:12.205 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.256 INFO:tasks.ceph.mon.h:Stopped 2023-09-29T23:19:12.256 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2023-09-29T23:19:12.257 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:19:12.309 INFO:tasks.ceph.mon.i:Stopped 2023-09-29T23:19:12.309 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-29T23:19:12.309 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-29T23:19:12.340 INFO:teuthology.orchestra.run.smithi012.stdout:1696029212.7490547 mgr.x (mgr.4111) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:19:12.341 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-29T23:19:12.341 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-29T23:19:12.411 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-29T23:19:12.482 INFO:teuthology.orchestra.run.smithi012.stdout:1696029212.7490547 mgr.x (mgr.4111) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:19:12.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:19:12.482 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-29T23:19:12.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:19:12.656 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-29T23:19:12.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:19:12.796 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-29T23:19:12.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:19:12.867 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-29T23:19:13.050 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:19:13.050 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-29T23:19:13.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:19:13.155 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-29T23:19:13.220 INFO:tasks.ceph:Archiving mon data... 2023-09-29T23:19:13.221 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.a.tgz 2023-09-29T23:19:13.221 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T23:19:13.236 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.RBgkEIV0WM 2023-09-29T23:19:13.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RBgkEIV0WM 2023-09-29T23:19:13.357 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.RBgkEIV0WM 2023-09-29T23:19:13.440 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.RBgkEIV0WM is 232KB 2023-09-29T23:19:13.532 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.RBgkEIV0WM 2023-09-29T23:19:13.546 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.b.tgz 2023-09-29T23:19:13.547 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T23:19:13.600 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.uwRWi3hfXr 2023-09-29T23:19:13.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uwRWi3hfXr 2023-09-29T23:19:13.704 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.uwRWi3hfXr 2023-09-29T23:19:13.786 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.uwRWi3hfXr is 233KB 2023-09-29T23:19:13.850 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.uwRWi3hfXr 2023-09-29T23:19:13.865 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.c.tgz 2023-09-29T23:19:13.865 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T23:19:13.919 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.yJeu2Sqyk5 2023-09-29T23:19:13.919 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yJeu2Sqyk5 2023-09-29T23:19:14.022 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.yJeu2Sqyk5 2023-09-29T23:19:14.103 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.yJeu2Sqyk5 is 233KB 2023-09-29T23:19:14.196 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.yJeu2Sqyk5 2023-09-29T23:19:14.211 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.d.tgz 2023-09-29T23:19:14.211 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T23:19:14.265 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.43uoU1YwAY 2023-09-29T23:19:14.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.43uoU1YwAY 2023-09-29T23:19:14.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.43uoU1YwAY 2023-09-29T23:19:14.450 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.43uoU1YwAY is 233KB 2023-09-29T23:19:14.542 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.43uoU1YwAY 2023-09-29T23:19:14.558 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.e.tgz 2023-09-29T23:19:14.559 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-09-29T23:19:14.612 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.ifMNrEQ6so 2023-09-29T23:19:14.613 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.ifMNrEQ6so 2023-09-29T23:19:14.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.ifMNrEQ6so 2023-09-29T23:19:14.797 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.ifMNrEQ6so is 233KB 2023-09-29T23:19:14.861 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.ifMNrEQ6so 2023-09-29T23:19:14.875 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.f.tgz 2023-09-29T23:19:14.875 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-09-29T23:19:14.893 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.DFAowNSvXO 2023-09-29T23:19:14.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.DFAowNSvXO 2023-09-29T23:19:15.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.DFAowNSvXO 2023-09-29T23:19:15.102 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.DFAowNSvXO is 233KB 2023-09-29T23:19:15.198 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.DFAowNSvXO 2023-09-29T23:19:15.213 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.g.tgz 2023-09-29T23:19:15.213 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-09-29T23:19:15.268 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.pqte2zQTas 2023-09-29T23:19:15.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.pqte2zQTas 2023-09-29T23:19:15.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.pqte2zQTas 2023-09-29T23:19:15.451 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.pqte2zQTas is 233KB 2023-09-29T23:19:15.546 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.pqte2zQTas 2023-09-29T23:19:15.560 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.h.tgz 2023-09-29T23:19:15.560 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-09-29T23:19:15.614 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.It9MYAlBxc 2023-09-29T23:19:15.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.It9MYAlBxc 2023-09-29T23:19:15.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.It9MYAlBxc 2023-09-29T23:19:15.798 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.It9MYAlBxc is 233KB 2023-09-29T23:19:15.891 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.It9MYAlBxc 2023-09-29T23:19:15.905 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/data/mon.i.tgz 2023-09-29T23:19:15.905 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-09-29T23:19:15.958 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.gQMIMrNjYs 2023-09-29T23:19:15.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.gQMIMrNjYs 2023-09-29T23:19:16.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.gQMIMrNjYs 2023-09-29T23:19:16.140 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.gQMIMrNjYs is 233KB 2023-09-29T23:19:16.233 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.gQMIMrNjYs 2023-09-29T23:19:16.247 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-29T23:19:16.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:19:16.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:19:16.310 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-29T23:19:16.310 DEBUG:teuthology.orchestra.run.smithi012:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:19:16.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:19:16.344 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:19:16.375 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:19:16.380 INFO:tasks.ceph:Archiving crash dumps... 2023-09-29T23:19:16.382 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi012/crash 2023-09-29T23:19:16.383 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:19:16.417 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi153/crash 2023-09-29T23:19:16.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:19:16.448 INFO:tasks.ceph:Compressing logs... 2023-09-29T23:19:16.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:19:16.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:19:17.864 INFO:tasks.ceph:Archiving logs... 2023-09-29T23:19:17.864 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi012/log 2023-09-29T23:19:17.865 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:19:18.176 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi153/log 2023-09-29T23:19:18.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:19:18.322 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:19:18.384 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f6551e2606c54e01a508d5ac2cf0a061 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:19:18.386 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-29T23:19:18.401 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-29T23:19:18.423 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-29T23:19:18.424 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:19:18.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:19:18.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-29T23:19:18.522 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-29T23:19:18.523 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:19:18.525 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:19:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:bash: ntpq: command not found 2023-09-29T23:19:18.538 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2023-09-29T23:19:18.542 INFO:teuthology.orchestra.run.smithi153.stdout:210 Number of sources = 4 2023-09-29T23:19:18.542 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:19:18.542 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -220us[ -205us] +/- 85ms 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -430us[ -415us] +/- 50ms 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +803us[ +817us] +/- 72ms 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi012.stdout:210 Number of sources = 4 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:19:18.543 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-09-29T23:19:18.544 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -211us[ -198us] +/- 85ms 2023-09-29T23:19:18.544 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -425us[ -412us] +/- 50ms 2023-09-29T23:19:18.544 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +830us[ +830us] +/- 72ms 2023-09-29T23:19:18.544 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:19:18.544 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-29T23:19:18.555 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-29T23:19:18.556 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-29T23:19:18.572 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-29T23:19:18.679 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-29T23:19:18.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:19:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:19:18.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:19:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:19:18.829 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-29T23:19:18.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-29T23:19:18.858 INFO:teuthology.task.internal:Duration was 1130.024167 seconds 2023-09-29T23:19:18.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-29T23:19:18.872 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-29T23:19:18.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:19:18.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:19:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:19:18.919 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:19:19.297 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-29T23:19:19.297 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2023-09-29T23:19:19.297 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:19:19.317 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2023-09-29T23:19:19.318 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:19:19.337 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-29T23:19:19.337 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:19:19.360 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:19:19.416 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-29T23:19:19.417 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:19:19.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:19:19.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-29T23:19:19.538 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-29T23:19:19.539 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:19:19.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:19:19.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-29T23:19:19.602 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:19:19.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:19:19.629 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2023-09-29T23:19:19.653 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2023-09-29T23:19:19.667 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:19:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:19:19.698 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:19:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:19:19.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-29T23:19:19.731 INFO:teuthology.task.internal:Transferring archived files... 2023-09-29T23:19:19.731 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi012 2023-09-29T23:19:19.731 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:19:19.777 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532/remote/smithi153 2023-09-29T23:19:19.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:19:19.812 INFO:teuthology.task.internal:Removing archive directory... 2023-09-29T23:19:19.813 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:19:19.815 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:19:19.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-29T23:19:19.872 INFO:teuthology.task.internal:Not uploading archives. 2023-09-29T23:19:19.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-29T23:19:19.914 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-29T23:19:19.915 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:19:19.917 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:19:19.930 INFO:teuthology.orchestra.run.smithi012.stdout: 262176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:19 /home/ubuntu/cephtest 2023-09-29T23:19:19.932 INFO:teuthology.orchestra.run.smithi153.stdout: 262556 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:19 /home/ubuntu/cephtest 2023-09-29T23:19:19.933 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-29T23:19:19.952 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-29T23:19:19.982 INFO:teuthology.nuke:Checking targets against current locks 2023-09-29T23:19:20.009 DEBUG:teuthology.nuke:shortname: smithi012 2023-09-29T23:19:20.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:19:20.032 DEBUG:teuthology.nuke:shortname: smithi153 2023-09-29T23:19:20.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:19:20.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:53:22.153335', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCXEjO1dqSmbvU7ggLljVXfP71JORi5hJrFaY3B58mHbBN58u4n1UxhMi8w6e3hO+p5ubHQ/W04fNy2DD5Xew='} 2023-09-29T23:19:20.061 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:53:22.151310', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEM5Ub7/Hqh7BEtkSEpSolDqKk4UjAfv947YCtF6f6+zrXTNFwNmC2cxGfrRNbRbOoq17u1in5gC8LzFf0TVXvs='} 2023-09-29T23:19:20.087 INFO:teuthology.orchestra.console.smithi012:Power off 2023-09-29T23:19:20.087 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:19:20.111 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2023-09-29T23:19:20.111 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:20.128 INFO:teuthology.orchestra.console.smithi153:Power off 2023-09-29T23:19:20.129 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:19:20.138 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-09-29T23:19:20.151 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2023-09-29T23:19:20.151 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:20.172 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-09-29T23:19:24.139 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:24.174 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:24.250 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-09-29T23:19:24.283 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-09-29T23:19:28.251 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:28.285 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:28.362 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-09-29T23:19:28.395 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2023-09-29T23:19:28.495 INFO:teuthology.orchestra.console.smithi153:Power off completed 2023-09-29T23:19:32.365 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:19:32.475 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2023-09-29T23:19:32.576 INFO:teuthology.orchestra.console.smithi012:Power off completed 2023-09-29T23:19:32.680 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} duration: 1130.024167060852 failure_reason: '"1696029212.7490547 mgr.x (mgr.4111) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f6551e2606c54e01a508d5ac2cf0a061 status: fail success: false 2023-09-29T23:19:32.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T23:19:32.765 INFO:teuthology.run:FAIL